Systems and methods for augmented neurologic rehabilitation

ABSTRACT

A method and system for augmented neurologic rehabilitation (ANR) of a patient is disclosed. The ANR system generates rhythmic auditory stimulus (RAS) and a visual augmented reality (AR) scene that are synchronized according to a common beat tempo and output to the patient during a therapy session. Sensor worn by the patient capture biomechanical data relating to repetitive movements performed by the patient in sync with the AR visual content and RAS. A critical thinking algorithm analyzes the sensor data to determine a spatial and temporal relationship of the patient&#39;s movements relative to the visual and audio elements and determine a level of entrainment of the patient and progression toward clinical/therapeutic goals. Additionally, a 3D AR modelling module configures the processor to dynamically adjust the augmented-reality visual and audio content output to the patient based on the determined level of entrainment and whether a training goal has been achieved.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims benefit of and priority to U.S.Provisional Patent Application No. 63/054,599, titled “Systems andMethods for Augmented Neurologic Rehabilitation,” to McCarthy et al.,filed Jul. 21, 2020, and further is a continuation-in-part of U.S.patent application Ser. No. 16/569,388 for “Systems and Methods forNeurologic Rehabilitation,” to McCarthy et al., which is a continuationof U.S. Pat. No. 10,448,888, titled, “Systems and Methods for NeurologicRehabilitation,” issue date Oct. 22, 2019, which is based on and claimspriority to U.S. Provisional Patent Application No. 62/322,504 filed onApr. 14, 2016, entitled “Systems and Methods for NeurologicRehabilitation,” which are each hereby incorporated by reference as ifset forth in their respective entireties herein.

TECHNICAL FIELD

The present disclosure relates generally to systems and methods forrehabilitation of a user having a physical impairment by providing musictherapy.

BACKGROUND

Many controlled studies over the past decade have emphasized theclinical role of music in neurologic rehabilitation. For example,regimented music therapy is known to directly enable cognition, motorand language enhancement. The process of listening to music enhancesbrain activity in many forms, igniting a widespread bilateral network ofbrain regions related to attention, semantic processing, memory,cognition, motor function and emotional processing.

Clinical data supports music therapy enhancing memory, attention,executive function, and mood. PET scan research on the neural mechanismsbehind music revealed that pleasant music can stimulate a widespreadnetwork between the cortical and subcortical region including theventral striatum, nucleus accumbens, amygdala, insula, hippocampus,hypothalamus, ventral tegmental area, anterior cingulate, orbitofrontalcortex, and ventral medial prefrontal cortex. The ventral tegmental areaproduces dopamine and has a direct connection to the amygdala,hippocampus, anterior cingulate and prefrontal cortex. Thismesocorticolimbic system, which can be activated by music, plays acritical role in mediating arousal, emotion, reward, memory attention,and executive function.

Neuroscience research has revealed how the fundamental organizationprocesses for memory formation in music shares a mechanism with thenon-musical memory processes. The basis of phrase groupings,hierarchical abstractions, and musical patterns have direct parallels intemporal chunking principles for non-musical memory processes. Thisimplies that memory processes activated with music could translate andenhance non-musical processes.

Accordingly, there remains a need for improved devices, systems, andmethods for protecting the use of user identity and for securelyproviding personal information.

SUMMARY

In one aspect of the disclosed subject matter, a system for augmentedneurologic rehabilitation of a patient is provided. The system comprisesa computing system having a processor configured by software modulescomprising machine-readable instructions stored in a non-transitorystorage medium.

The software modules include an AA/AR modelling module that, whenexecuted by the processor, configures the processor to generate anaugmented-reality (AR) visual content and rhythmic auditory stimulus(RAS) for output to a patient during a therapy session. In particular,the RAS comprises beat signals output at a beat tempo and the AR visualcontent includes visual elements moving in a prescribed spatial andtemporal sequence based on the beat tempo.

The system further comprises an input interface in communication withthe processor for receiving real-time patient data includingtime-stamped biomechanical data of the patient relating to repetitivemovements performed by the patient in time with the AR visual contentand RAS. In particular, the biomechanical data is measured using asensor associated with the patient.

The software modules further include a critical thinking algorithm (CTA)module that configures the processor to analyze the time-stampedbiomechanical data to determine a temporal relationship of the patient'srepetitive movements relative to the visual elements and beat signalsoutput at the beat tempo to determine a level of entrainment relative toa target parameter. Moreover, the AA/AR modelling module furtherconfigures the processor to dynamically adjust the AR visual and RASoutput to the patient in synchrony and based on the determined level ofentrainment.

According to a further aspect, a method for augmented neurologicrehabilitation of a patient having a physical impairment is provided.The method is implemented on a computer system having a physicalprocessor configured by machine-readable instructions which, whenexecuted, perform the method.

The method includes the step of providing rhythmic auditory stimulus(RAS) for output to a patient via an audio output device during atherapy session. In particular, the RAS comprises beat signals output ata beat tempo.

The method also includes the step of generating augmented-reality (AR)visual content for output to a patient via an AR display device. Inparticular, the AR visual content includes visual elements moving in aprescribed spatial and temporal sequence based on the beat tempo andoutput in synchrony with the RAS. The method further includes the stepof instructing, the patient to perform repetitive movements in time withthe beat signals of the RAS and corresponding movement of the visualelements of the AR visual content.

The method further includes the step of receiving real-time patient dataincluding time-stamped biomechanical data of the patient relating torepetitive movements performed by the patient in time with the AR visualcontent and RAS. In particular, the biomechanical data is measured usinga sensor associated with the patient.

The method further includes the step of analyzing the time-stampedbiomechanical data to determine a temporal relationship of the patient'srepetitive movements relative to the visual elements and beat signalsoutput according to the beat signal to determine an entrainmentpotential. Additionally, the method includes the steps of dynamicallyadjusting the AR visual content and RAS for output to the patient insynchrony and based on the determined entrainment potential not meetinga prescribed entrainment potential and continuing the therapy sessionusing the adjusted AR visual content and RAS.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the devices,systems, and methods described herein will be apparent from thefollowing description of particular embodiments thereof, as illustratedin the accompanying drawings. The drawings are not necessarily to scale,emphasis instead being placed upon illustrating the principles of thedevices, systems, and methods described herein.

FIG. 1 is a diagram illustrating a system for therapy of a user byproviding music therapy in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 2 is a diagram illustrating several components of a system forrehabilitation of a user by providing music therapy in accordance withexemplary embodiments of the disclosed subject matter;

FIG. 3 is a schematic drawing of a sensor for measuring thebiomechanical movements of a patient in accordance with exemplaryembodiments of the disclosed subject matter;

FIG. 4 is a diagram illustrating several components of the system inaccordance with exemplary embodiments of the disclosed subject matter;

FIG. 5 illustrates an exemplary display of a component of a system forrehabilitation of a user by providing music therapy in accordance withexemplary embodiments of the disclosed subject matter;

FIG. 6 is a flow diagram for one implementation of an analytics processin accordance with exemplary embodiments of the disclosed subjectmatter;

FIGS. 7-10 are flow diagrams for one implementation of a process inaccordance with exemplary embodiments of the disclosed subject matter;

FIG. 11 is a time plot illustrating music and physical movement of thepatient in accordance with exemplary embodiments of the disclosedsubject matter;

FIGS. 12-13 illustrate a patient response in accordance with exemplaryembodiments of the disclosed subject matter;

FIGS. 14-15 illustrate a patient response in accordance with exemplaryembodiments of the disclosed subject matter;

FIGS. 16-17 illustrate a patient response in accordance with exemplaryembodiments of the disclosed subject matter;

FIG. 18 illustrates an implementation of a technique for gait trainingof a patient in accordance with exemplary embodiments of the disclosedsubject matter;

FIG. 19 illustrates an implementation of a technique for neglecttraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 20 illustrates an implementation of a technique for intonationtraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 21 illustrates an implementation of a technique for musicalstimulation training of a patient in accordance with exemplaryembodiments of the disclosed subject matter;

FIG. 22 illustrates an implementation of a technique for gross motortraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 23 illustrates an implementation of a technique for grip strengthtraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 24 illustrates an implementation of a technique for speech cueingtraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 25 illustrates an implementation of a technique for training of aminimally conscious patient in accordance with exemplary embodiments ofthe disclosed subject matter;

FIGS. 26-28 illustrates an implementation of a technique for attentiontraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 29 illustrates an implementation of a technique for dexteritytraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 30 illustrates an implementation of a technique for oral motortraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 31 illustrates an implementation of a technique for respiratorytraining of a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 32 is a diagram illustrating an augmented neurologicrehabilitation, recovery or maintenance (“ANR”) system for providingtherapy to a patient in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 33 is a graphical visualization of measured parameters, systemresponses and target/goal parameters during a therapy session performedusing the ANR system of FIG. 32 in accordance with exemplary embodimentsof the disclosed subject matter;

FIG. 34 is a graph depicting exemplary results relating to metabolicchange resulting from a training session performed using the ANR systemof FIG. 32 in accordance with exemplary embodiments of the disclosedsubject matter;

FIG. 35 is an exemplary augmented reality (AR) display generated by theANR system for display to the patient during a therapy session inaccordance with exemplary embodiments of the disclosed subject matter;

FIG. 36A is an exemplary AR display generated by the ANR system fordisplay to the patient during a therapy session in accordance withexemplary embodiments of the disclosed subject matter;

FIG. 36B is an exemplary AR display generated by the ANR system fordisplay to the patient during a therapy session in accordance withexemplary embodiments of the disclosed subject matter;

FIG. 37 illustrates an implementation of a technique for gait trainingby providing augmented audio and visual stimulus to a patient inaccordance with exemplary embodiments of the disclosed subject matter;

FIG. 38 is a hybrid system and process diagram conceptually illustratingthe ANR system configured for implementing the gait-training techniquein accordance with exemplary embodiments of the disclosed subjectmatter;

FIG. 39 is a hybrid system and process diagram conceptually illustratingthe augmented audio (AA) device component of the ANR system of FIG. 38in greater detail in accordance with exemplary embodiments of thedisclosed subject matter;

FIG. 40 is an exemplary AR display generated by the ANR system fordisplay to the patient during a therapy session in accordance withexemplary embodiments of the disclosed subject matter; and

FIG. 41 is an exemplary AR display generated by the ANR system fordisplay to the patient during a therapy session in accordance withexemplary embodiments of the disclosed subject matter;

DETAILED DESCRIPTION

The present invention relates generally to systems, methods andapparatus for implementing a dynamic closed-loop rehabilitation platformsystem that monitors and directs human behavior and functional changes.Such changes are in language, movement, and cognition that aretemporally triggered by musical rhythm, harmony, melody, and force cues.

In various embodiments of the invention, a dynamic closed-looprehabilitation platform music therapy system 100 is provided illustratedin FIG. 1, which includes sensor components and systems 102,edge-processing components 104, collector components 106, analyticssystems 108, and music therapy center 110. As will described in greaterdetail below, the sensor components, edge processing components,collector components machine learning processes and music therapy centermay be provided on various hardware components. For example, in oneembodiment, the sensor components and edge processing components may belocated or worn by the patient. In such embodiments, the collectorcomponents and music therapy center may be provided on a handhelddevice. In such embodiments the analytics systems may be located on aremote server.

Sensor Systems

Throughout the description herein, the term “patient” is used to referto the individual receiving musical therapy treatment. The term“therapist” is used to refer to the individual providing musical therapytreatment. In some embodiments, the patient is able to interact withthis system described herein without the presence of the therapist toadminister the treatment.

The sensor components 102 provide sensed biomechanical data about thepatient. In some embodiments, the sensor components can include (1)wearable wireless real-time motion sensing devices or IMU (inertialmeasurement units), (2) wearable wireless real-time combination multiplezone foot plantar pressure/6-dimensional motion capture (IMU) devices,such as sensor 200, (3) wearable wireless real-time Electromyogram (EMG)devices, such as sensor 208 and (4) real-time wireless near infrared(NIR) video capture devices, such as imaging device 206 (See FIG. 4).

As illustrated in FIG. 2, the systems and methods described herein areused in connection with treating walking disorders of the patient.Accordingly, the exemplary sensor 200 can be a combination multiple zonefoot plantar pressure/6-degrees of freedom motion capture device. Sensor200 records the patient's foot pressure and 6-degrees of freedom motionprofile while the patient walks during a music therapy session. In someembodiments, the foot pressure/6-degrees of freedom motion capturedevice has variable recording duration intervals with a sampling rate of100 Hz for a foot pressure profile that comprises 1 to 4 zones resultingin 100 to 400 pressure data points per foot per second.

The sensor 200 can include a foot pressure pad 202 having a heel pad(for measuring one zone of pressure, e.g., heel strike pressure) to afull insole pad (for measuring 4 zones of pressure). The pressuremeasurements are made by sensing the resistive changes in transducermaterial as a result of the compression due to the patient's weighttransferred to the foot. These foot pressure maps are obtained for eachsampling interval or at specific instants during a music therapysession.

The sensor 200 can include a 6-Dimensional motion capture device 204that detects the changes in motion via a 6-degrees of freedomMicro-Electro-Mechanical Systems (MEMS) based sensor which determineslinear acceleration in 3 dimensions, A_(x), A_(y), A_(z) and rotationalmotion as pitch, yaw, and roll. Sampling at 100 Hz will produce 600motion data points per second. These foot motion captures are obtainedfor each sampling interval or at specific instants during a musictherapy session.

The multiple zone pressure sensing with the 6-degrees of freedom motioncapture device allows for map-able spatial and temporal gait dynamicstracking while walking. A schematic diagram of the sensor 200 isillustrated in FIG. 3.

From a system perspective, as illustrated in FIG. 4, the patient P usestwo foot sensors 200, one for each foot designated as Right 200R andLeft 200L. In an exemplary embodiment, the right foot sensor 200Rwirelessly communicates time-stamped internal measurement unit data andheel strike pressure data over a first channel, e.g., channel 5, in theIEEE 802.15.4 direct sequence spread spectrum (DSSS) RF band. The leftfoot sensor 200L wirelessly communicates time-stamped internalmeasurement unit data and heel strike pressure data over a secondchannel, e.g., channel 6, in the IEEE 802.15.4 direct sequence spreadspectrum (DSSS) RF band. A tablet or laptop 220, optionally used by thetherapist T, as described below, includes a wireless USB hub containingtwo IEEE 802.15.4 DSSS RF transceivers tuned to the first and secondchannels, e.g., channel 5 and 6, in order to capture the right/left footsensor RF data. A handheld wireless trigger 250 is used to start andstop video and/or to make notations and index the time stream asdiscussed in greater detail below.

A video analytics domain can be used to extract patient semantic andevent information about therapy sessions. Patient actions andinteractions are components in the therapy that affect the therapycontext and regiment. In some embodiments, one or more image capturedevices 206, such as video cameras, (see FIG. 4) are used with atime-synched video feed. Any appropriate video may be incorporated intothe system to capture patient movement; however, Near Infrared (NIR)video capture is useful to preserve the patient's privacy and to reducethe video data to be processed. The NIR video capture device capturesNIR video images of a patient's body such as the position of thepatient's torso and limbs. Further, it captures the patient's real-timedynamic gait characteristics as a function of a music therapy session.In some embodiments, the video is captured with a stationary camera, inwhich the background is subtracted to segment out foreground pixels.

As illustrated in FIG. 4, the one or more video cameras 206 aretriggered by the tablet or laptop application when therapy sessionstarts. The video cameras 206 can be stopped or started by a hand heldwireless trigger unit 250 by the therapist. This allows for labeledtime-stamped index to be created in the captured biomechanical sensordata and video data streams.

In some embodiments, wearable wireless real-time Electromyogram (EMG)devices 208 can be worn by the patient. EMG sensors provide the entirebi-ped profile for major muscle firing for locomotion. Such sensorsprovide data regarding the exact time when the muscle are fired.

Edge Processing

In some embodiments, the edge process is performed at the sensors 200,where sensor data is captured from the IMU and pressure sensors. Thissensor data is filtered, grouped into various array sizes for furtherprocessing into frames reflecting extracted attributes and features, andwhere these frames are sent, e.g., wirelessly, to the collector 106 on atablet or laptop. It is understood that the raw biomechanical sensordata obtained from the sensor 200 can alternatively be transferred to aremote processor for the collect for the edge processing functions totake place.

The wearable sensors 200, 208 and the video capture devices 206,generates sensor data streams that are processed holistically tofacilitate biomechanical feature extraction and classification. Sensorfusion, combining the outputs from multiple sensors capturing a commonevent, better captures a result than any single constituent sensorinputs.

Capturing patient activities in the music therapy context formalizes theinteractions as applied to the music therapy and in developing patientspecific and generalized formal indicators of the music therapyperformance and efficacy. Extracting video features and then analyzingallows for the capture of semantic, high-level information about patientbehaviors.

In processing video, a learned background subtraction technique is usedto create a background model which incorporates any variation inlighting conditions and occlusions in the physical area where musictherapy occurs. The result of the background subtraction is a binaryforeground map with an array of foreground blobs which are twodimensional contours. Thus, the video is sliced into individual imageframes for future image processing and sensor fusion. Video informationis provided with additional meta data by merging in the edge-processedsensor data from the IMU, foot pressure pad(s), and EMG sensors. Thesensor data can be time synched with the other data using the RFtrigger. Data can be sent directly to the collector, stored on thememory of the internal board, or analyzed on the edge running the OpenCVlibrary.

The edge processor 104 can be a microprocessor, such as a 32-bitmicroprocessor incorporated into the foot pressure/6-degrees of freedommotion capture device that enables fast multiple zone scanning at a rateof 100 to 400 complete foot pressure/6-degrees of freedom motionprofiles per second.

The foot pressure/6-degrees of freedom motion capture device collectsfoot pressure/6-degrees of freedom motion profile data for real-timegait analysis resulting in feature extraction and classification. Insome embodiments, the foot pressure/6-degrees of freedom motion capturedevice initializes an micro controller unit (MCU), continuous operatorprocess (COP), general purpose input output (GPIO), serial peripheralinterface (SPI), interrupt request (IRQ), and sets a desired RFtransceiver clock frequency by calling routines including microcontroller unit initialize (MCUInit), general purpose input outputinitialize (GPIOInit), serial peripheral interface initialize (SPIInit),interrupt request acknowledgeinitialize (IRQInit), interrupt requestacknowledge (IRQACK), Serial Peripheral Interface Driver Read(SPIDrvRead), and IRQPinEnable. MCUInit is the master initializationroutine which turns off the MCU watchdog and sets the timer module touse bus clock (BUSCLK) as a reference with a pre-scaling of 32.

The state variable gu8RTxMode is set to SYSTEM_RESET_MODE and theroutines GPIOInit, SPIInit and IRQInit are called. The state variablegu8RTxMode is set to RF_TRANSCEIVER_RESET_MODE and the IRQFLAG ischecked to see if IRQ is asserted. The RF transceiver interrupts arefirst cleared using SPIDrvRead, then the RF transceiver is checked forATTN IRQ interrupts. Lastly, for MCUInit, calls are made to PLMEPhyResetto reset the physical MAC layer, IRQACK (to ACK the pending IRQinterrupt) and IRQPinEnable which is to pin, Enable, IE, and IRQ CLR, onsignal's negative edge.

The foot pressure/6-degrees of freedom motion sensor 200 will wait for aresponse from the foot pressure/6-degrees of freedom motion collectingnode, e.g., 250 milliseconds, to determine whether a default full footpressure scan will be done or a mapped foot pressure scan will beinitiated. In the case of a mapped foot pressure scan, the footpressure/6-degrees of freedom motion collecting node will send theappropriate electrode the foot pressure scan mapping configuration data.

One aspect of the analytics pipeline is the feature set engineeringprocess which will define those captured sensor values and theirresulting sensor-fused values that are used to create feature vectors todefine the input data structures for the analytics. Representativevalues are Ax(i), Ay(i), Az(i), and Ph(i), where i is the ith sample,where Ax(i) is the acceleration in the x-direction which is Lateral inrelation to the foot sensor; Ay(i) is the acceleration in they-direction which is Front in relation to the foot sensor; Az(i) is theacceleration in the z-direction which is Up in relation to the footsensor; and Ph(i) is the heel strike pressure. The Sensor values arepresented in Table 1:

TABLE 1 Avg (Ax) = Sum [Ax(i) over i = 0 to i = N]/N Avg (Ax) = Sum[Ax(i) over i = 0 to i = N]/N Avg (Ay) = Sum [Ay(i) over i = 0 to i =N]/N Avg (Az) = Sum [Az(i) over i = 0 to i = N]/N Max (Ax) in the rangeof Ax(i) from i = 0 to i = N Max (Ay) in the range of Ay(i) from i = 0to i = N Max (Az) in the range of Az(i) from i = 0 to i = N Min (Ax) inthe range of Ax(i) from i = 0 to i = N Min (Ay) in the range of Ay(i)from i = 0 to i = N Min (Az) in the range of Az(i) from i = 0 to i = NAvg (Ph) = Sum [Ph(i) over i = 0 to i = N]/N Max (Ph) in the range ofPh(i) from i = 0 to i = N where N = window size

In some embodiments, the sensor-fused technique uses the heel strikepressure value Ph(i) to “gate” the analysis of the following exemplaryfeature values to derive a window of data as will be described below.For example, the “onset” (start) can be determined based on heelpressure exceeding a threshold indicating heel strike, and the “stop”based on heel pressure falling below a threshold indicating heel off,presented in Table 2, below. It is understood, that heel strike pressureis one example of a parameter that can be used to for the “gate”analysis. In some embodiments, “gating” is determined by use of IMUsensor data, video data, and/or EMG data.

TABLE 2 Power Factor PF(i) = Sqrt (Ax(i)**2 + Ay(i)**2 + Az(i)**2)Windowed Total Motion Intensity = [Avg(Ax) + Avg(Ay) + Avg(Az)]/3Windowed Lateral Tremor Intensity = Sum [ (Ax(i) − Ax(i + 1))**2] from i= 0 to i = N Windowed Total Tremor Intensity = Sum [ (Ax(i) − Ax(i +1))**2] + Sum [ (Ay(i) − Ay(i + 1))**2] + Sum [ (Az(i) − Az(i + 1))**2]from i = 0 to i = N Windowed Differential Ax = Max (Ax) − Min (Ax)Windowed Differential Ay = Max (Ay) − Min (Ay) Windowed Differential Az= Max (Az) − Min (Az) where N = window size

Higher level feature values are calculated from the fused sensor values,such as exemplary values presented in Table 3:

TABLE 3 Step Count (Total number) Step Length Right (centimeters-cm)Step Length Left (cm) Step Time Right (milliseconds-msec) Step Time Left(msec) Asymmetry Factor Right/Left Step Time (Step Time Right-Step TimeLeft) Step Width (cm) Cadence (strides per minute) Stride Length (cm)Stride Velocity (cm/sec) Stride Time Right (msec) Stride Time Left(msec) Asymmetry Factor Right/Left Stride Time (Stride Time Right-StrideTime Left) Stride Tremor (Windowed Lateral Tremor Intensity) StrideFluidity (Windowed Total Tremor Intensity) Stride Tremor Accumulated(Windowed Lateral Tremor Intensity) Stride Fluidity Accumulated(Windowed Total Tremor Intensity) Swing Time Right Foot (msec) SwingTime Left Foot (msec) Stance Phase Right Foot (msec) Stance Phase LeftFoot (msec) Asymmetry Factor Stance Phase Right/Left Stance Phase(Stance Phase Right-Stance Phase Left) Double Support Stance Time (msec)Vertical Displacement [Mid-Stance] Max (cm) Vertical Displacement[Double Support] Min (cm) Heel Strike Time Right Foot (msec) Heel StrikeTime Left Foot (msec) Heel Strike Pressure Right Foot (shift N-Newton)Heel Strike Pressure Left Foot (N) Asymmetry Factor Right/Left HeelStrike Pressure (Heel Strike Pressure Right Foot-Heel Strike PressureLeft Foot) Distance Travelled Accumulated (meters-m) Average Velocity(m/min) Variability of each of the factors

The system described herein provides the capability to “gate” or providea “window” with respect to the patient biomechanical data. Gating of thebiomechanical data is useful for repetitive patient movements, such asthe repetitive strides while a patient is walking. Sensor data, from oneor more sources, such as pressure sensors, IMU sensors, video data, andEMG data, is used to identify cycles of movement that repeat over time.For example, when a patient walks, foot pressure increases and decreasesrepetitively, as the patient's foot contacts the ground and then islifted off the ground. Likewise, the velocity of the foot increase asthe foot moves forward and decreases to zero while the foot is plantedon the ground. As a further example, the Y-position or height of thepatient's foot cycles between a low position (on the ground) and a highposition (approximately in mid stride). The “gating” techniqueidentifies repeating cycles or “windows” within such data. In the caseof a patient walking, the cycle is repeated with each step. Althoughthere may be variations between cycles, e.g., between steps, certainpatterns repeat with each cycle. Selecting an onset time (start time) ofeach cycle involves locating an identifiable point (maximum or minimum)of a biomechanical parameter. The selection of the parameter for theonset time is selected based upon the available data. Thus, in someembodiments, the moment when the heel-strike pressure exceeds athreshold may be used to demarcate the onset time of each cycle. (See,e.g., FIG. 5. Pressure 316 a and 316 b includes a cyclic characteristic.“Onset” may be determined at the moment the pressure exceeds athreshold.) Similarly, the onset time may be demarcated when footvelocity falls to zero.

In some embodiments, raw frames data is pre-processed, taking theinstant data and “gating” it, e.g., identifying a window, and thenanalyzing data within that window to identify outliers and to performanalysis on the data, e.g., exponential analysis, averaging data amongmultiple windows. Fusion of sensor data, by including both IMU data andheel-strike pressure data, allows for more precise identification ofonset times for a single stride or other repeated units of motion thanusing data from a single sensor. Sensor data captured within a singlestride is considered a “window,” and information extracted from thisanalysis includes, e.g., stride length, step count, cadence, time whenstep occurs, distance traveled, stance phase/swing phase, double supporttime, velocity, symmetry analysis (e.g., between left and right leg),outward swing, shuffling, power vector, lateral acceleration, stepwidth, variability of each of these dimensions, additional parametersderived from the above-described information, etc. Feature extractioncan be processed on microprocessor chip, e.g., a 32-bit chip. Capture ofwireless synchronous-gated biomechanical sensor data and video datacapture capability allows for time-series template creation.

The data can be indexed by the patient or the therapist during a musictherapy session. The “gating” functionality described above is useful totie exception conditions to particular strides or steps. For example,the therapist may observe a particular exception condition or behavior(such as an anomaly or incident) in the patient's movement. The indexingfunction allows the therapist to initiate, such as, capture to “record,”an exception condition or behavior via a user interface on the handheldtablet or laptop, such as the wireless trigger unit 250 illustrated inFIG. 4, or voice control. A notation can be created that includes atimestamp and a comment, such as the occurrence of a “stumble” by thepatient while walking. Such indexing facilitates time-series templatecreation. These time-series templates will be studied for review oftherapy session events and for the development of times-series templatesfor training machine learning algorithms such as non-linearmulti-layered perceptrons (NLMLP), convolutional neural networks (CNNs),and recurrent neural networks (RNNs) with long short term memory (LSTM).

In one embodiment, a communication protocol is provided to transfersensor data from edge processing 104 (e.g. at the sensors 200) to thecollector 106. See Table 4 below. In some embodiments, if the connectionis idle for more than 100 ms, the RF has timed out.

TABLE 4 [0x10] Start of frame [0x49] FootClipSensor ID = ‘I’ [0x52] or[0x4C] Which FootClipSensor = ‘R’ or ‘L’ [0x00~0xFF] Zone 1 [0x00~0xFF]Zone 2 [0x00~0xFF] Zone 3 [0x00~0xFF] Zone 4 [Az] Az [Ay] Ay [Ax] Ax[HighByteSeqNum] High Byte Sequence [LowByteSeqNum] Low Byte Sequence

In one embodiment, the foot pressure sensor zone scanning is performedby the FootScan routine where the FootDataBufferindex is initialized andthe foot pressure sensor zone is activated by enabling MCU directionmode for output [PTCDD_PTCDDN=Output] and bringing the associated portline low [PTCD_PTCD6=0]. As the foot pressure sensor zone is activatedbased on the foot pressure sensor zone scanning map, the foot pressuresensor zones attached to the MCU analog signal ports will be sampled andthen the current voltage reading converts them into digital form (whichis the-time zone foot pressure).

Several variables such as FootDataBufferindex and IMUBufferIndex areused to prepare the IEEE 802.15.4 RF packetsgsTxPacket.gau8TxDataBuffer[ ] which are for sending the data to be usedin FootDataBuffer[ ] and IMUBuffer[ ]. The RF packets are sent using theRFSendRequest(&gsTxPacket) routine. This routine checks to see ifgu8RTxMode is set at IDLE_MODE and uses gsTxPacket as a pointer to callthe RAMDrvWriteTx routine which then calls SPIDrvRead to read the RFtransceiver's TX packet length register contents. Using these contents,mask length settings update and then add 2 for CRC and 2 for code bytes.

SPISendChar is called to send a 0×7E byte, which is the 2nd code byteand then the SPIWaitTransferDone is called again to verify the send isdone. With these code bytes sent, then the rest of the packet is sentusing a for loop, where psTxPkt→u8DataLength+1 are the number ofiterations to a series of sequential to SPISendChar,SPIWaitTransferDone, SPIClearRecieveDataReg. When complete, the RFtransceiver is loaded with the packet to send. The ANTENNA_SWITCH is setto transmit, the LNA_ON mode enabled, and finally a RTXENAssert callmade to actually send the packet.

Collector

The primary function of the collector 106 is to capture data from theedge processing 104, transfer data to and receive processed data fromthe analytics system 108, and transfer data to the music therapy center110, described below. In some embodiments, the collector 106 providescontrol functionality, e.g., a user interface to login, configure thesystem, and interact with users, and includes a display unit tovisualize/display data. The collector 106 may include lightweightanalytics or machine learned algorithms for classification (e.g.,lateral tremor, asymmetry, instability, etc).

The collector 106 receives body, motion, and localization data from theedge processor 104. Data received at collector 106 can be raw orprocessed at the edge 104 prior to transfer to the collector. Forexample, the collector 106 receives fused sensor data, subject to“windowing” and feature extraction. The transferred data can include twolevels of data: (1) RF Packets sent from the Right/Left foot sensors asdescribed in Table 1, (2) RF Packets from the Right/Left foot sensorswhich contains higher level attributes and features as described inTables 2 and 3. The collector 106 locally stores the data. In someembodiments, the collector 106 classifies movement from the receiveddata, e.g., comparing it to models stored locally (pre-downloaded fromthe analytics system) or sent to analytics system for classification.The collector may include a display unit to visualize/display the data.

In some embodiments the collector 106 operates on a local computer thatincludes a memory, a processor and a display. Exemplary devices on whichthe collector is installed can include augmented reality (AR) devices,virtual reality (VR) devices, tablets, mobile devices, laptop computers,desktop computers, and the like. FIG. 2 illustrates a handheld device220 having a display 222, and which performs the collector functions. Insome embodiments, the connection parameters for transferring databetween the patient sensor and the collector are made include the use ofDevice Manager in Windows (e.g., Baud rate: 38400, data bits: 8; parity:none, stop bits: 1). In some embodiments, the collector 106 includes aprocessor that is held or worn by the music therapy patient. In someembodiments, the collector 106 includes a processor that is remote fromthe music therapy patient and carried by a therapist, and connectedwirelessly or via a wired connection to the music therapy patient.

In one embodiment, a foot pressure/6-degrees of freedom motioncollecting node captures RF transmitted data packets containingreal-time foot pressure/6-degrees of freedom motion profile data fromthe foot pressure/6-degrees of freedom motion capture device. This isstarted by the foot pressure/6-degrees of freedom motion collecting nodewhich creates a RF packet receive queue that is driven by a call backfunction on RF transceiver packet receive interrupts.

When an RF packet is received from a foot pressure/6-degrees of freedommotion capture device 200, a check is first made to determine if thisfrom a new foot pressure/6-degrees of freedom motion capture device oran existing one. If this is from an existing foot pressure/6-degrees offreedom motion capture device, RF packet sequence numbers are checked todetermine continuous synchronization before further analyzing thepacket. If this is a foot pressure capturing/6-degrees of freedom motiondevice, a foot pressure/6-degrees of freedom motion capture devicecontext state block is created and initialized. The context state blockincludes information, e.g., the foot pressure profile.

Above this RF packet session level process for node to nodecommunication, is the analysis of the RF packet data payload. Thispayload contains the foot pressure profile based on the current variablepressure following the 6-degrees of freedom motion. This is structuredas follows: |0×10|start|F1|F2⊕F3|F4|Ax|Ay|Az|Pi|Yi|Ri|XOR checksum|.

The IEEE 802.15.4 standard specifies a maximum packet size of 127 bytesand the Time Synchronized Mesh Protocol (TSMP) reserves 47 Bytes foroperation, leaving 80 Bytes for payload. The IEEE 802.15.4 is compliantwith the 2.4 GHz Industrial, Scientific, and Medical (ISM) band RadioFrequency (RF) transceiver.

The RF module contains a complete 802.15.4 Physical layer (PHY) modemdesigned for the IEEE 802.15.4 wireless standard which supportspeer-to-peer, star, and mesh networking. It is combined with a MCU tocreate the required wireless RF data link and network. The IEEE 802.15.4transceiver supports 250 kbps O-QPSK data in 5.0 MHz channels and fullspread-spectrum encode and decode.

In some embodiments, control, reading of status, writing of data, andreading of data is done through the sensing system node device's RFtransceiver interface port. The sensing system node device's MPUaccesses the sensing system node device's RF transceiver throughinterface ‘transactions’ in which multiple bursts of byte-long data aretransmitted on the interface bus. Each transaction is three or morebursts long, depending on the transaction type. Transactions are alwaysread accesses or write accesses to register addresses. The associateddata for any single register access is always 16 bits in length.

In some embodiments, control of the foot pressure/6-degrees of freedommotion collecting node's RF transceiver and data transfers areaccomplished by means of a Serial Peripheral Interface (SPI). Althoughthe normal SPI protocol is based on 8-bit transfers, the footpressure/6-degrees of freedom motion collecting collector node's RFtransceiver imposes a higher level transaction protocol that is based onmultiple 8-bit transfers per transaction. A singular SPI read or writetransaction consists of an 8-bit header transfer followed by two 8-bitdata transfers.

The header denotes access type and register address. The following bytesare read or write data. The SPI also supports recursive ‘data burst’transactions in which additional data transfers can occur. The recursivemode is primarily intended for Packet RAM access and fast configurationof the foot pressure/6-degrees of freedom motion collecting node's RF

In some embodiments, all foot pressure sensor zones are sequentiallyscanned and the entire process repeats until a reset condition orinactivity power-down mode. The 6-degrees of freedom motion is capturedby a serial UART interface to the Inertial Measurement Unit (IMU) fromthe MCU. The sampling rate for all sensing dimensions is 100-300 Hzwhich is Ax, Ay, Az, Pitch, Yaw, Roll and which sampled data is storedin IMUBuffer[ ].

A call is made to SPIDryWrite to update the TX packet length field.Next, a call to SPIClearRecieveStatReg is made to clear the statusregister followed by a call to SPIClearRecieveDataReg to clear thereceive data register to make the SPI interface ready for reading orwriting. With the SPI interface ready, a call is made to SPISendCharsending a 0xFF character which represents the 1st code byte and thenSPIWaitTransferDone is called to verify the send is done.

FIG. 5 is an exemplary output 300 that may be provided on display 222 ofthe handheld device. For example, when therapy is provided for apatient's gait, the display output 300 may include a portion for theright foot 302 and a portion for the left foot 304. As a function oftime, the display for the right foot includes accelerations A_(x) 310 a,A_(y) 312 a, and A_(z) 314 a, and foot pressure 316 a. Similarly, thedisplay for the left foot includes acceleration A_(x) 310 a, A_(y) 312a, and A_(z) 314 a, and foot pressure 316 a.

Classification is understood as the correlation of data, e.g., sensorfused data, feature data, or attribute data to real world events, e.g.,activities or disposition of the patient. Typically, the classificationis created and performed on the analytics system 108. In someembodiments, the collector 106 has a local copy of some ‘templates.’Thus, the incoming sensor data and feature extracted data can beclassified at the collector or the analytics system.

Context refers to the circumstances or facts that form the setting foran event, statement, situation, or idea. Context-aware algorithmsexamine the “who,” “what,” “when” and “where” related to the environmentand time in which the algorithm is executed against certain data. Somecontext-aware actions include an identity, location, time, and activitybeing executed. In using contextual information to formulate adeterministic action, context interfaces occur among the patient, theenvironment, and the music therapy session.

The patient's reaction context to a music therapy session can involve alayer of algorithms that interpret the fused sensor data to inferhigher-level information. These algorithms distill the patient reactioncontext. For example, a patient's bio-mechanical gait sequence isanalyzed as it relates to a specific portion of the music therapysession. In one example, “lateral tremor” is the classifier of interest.Accordingly, it is determined that the patient's gait becomes more fluidwith less lateral tremor.

Analytics Systems

The analytics systems 108, sometimes referred to as the back end system,store large models/archives and include machine learning/analyticsprocessing, with the models described herein. In some embodiments, a webinterface for login to view archived data, and a dashboard is alsoprovided. In some embodiments the analytics system 108 is located on aremote server computer which receives data from the collector 106running on a handheld unit such as handheld device or tablet 220. It iscontemplated that the processing capability needed to perform theanalytics and machine learning functions of the analytics system 108 maybe also located on the handheld device 220.

Data is transferred from the collector 106 to the analytics systems 108for analytics processing. As illustrated in FIG. 6, the analyticsprocessing 400 includes a user-interface 402 for receiving data from thecollector 106. A database storage 404 receives incoming data from thecollector 106 for storage. Training data as well as outputs of theanalytics processing, e.g., the ensemble machine learning system 410,may also be stored on storage 404 to facilitate the creation andrefinement of the predictive models and classifiers. A data bus 406allows flow of data through the analytics processing. A training process408 is performed on training data to derive one or more predictivemodels. An ensemble machine learning system 410 utilizes the predictivemodels. The output of the ensemble machine learning system 410 is anaggregation of these predictive models. This aggregated output is alsoused for classification requirements with template classifiers 412, suchas tremor, symmetry, fluidity, or learned biomechanical parameters suchas entrainment, initiation, etc. An API 418 connects to the collectorand/or music therapy Center. Therapy algorithms 414 and predictivealgorithms 416 include multi-layer perceptron neural networks, hiddenMarkov models, Radal based function networks, Bayesian inference models,etc.

An exemplary application of the systems and methods described herein isanalysis of a patient's bio-mechanical gait. The gait sequence isfeature-extracted into a series of characteristic features. The presenceof these and other features in captured sensor-fused data inform thecontext detection algorithm if the patient's bio-mechanical gaitsequence is valid. Bio-mechanical gait sequence capture requires robustcontext detection, which is then abstracted over a representativepopulation of music therapy patients.

An example of such an activity is the location of a patient at aninstance in time and their response to the music therapy at that time.The recognition and correlation of patient music therapy responsesallows for recognition specific patterns of music therapy patientresponses. Specific music therapy regimes are then benchmarked andanalyzed for performance and efficacy by creating a baseline of musictherapy patient responses and correlating them to future music therapypatient responses.

In combination with motion sensing, a distance metric with gaitbio-mechanics capture is used to determine patient path trajectory usingtemporal and spatial variations/deviations between two or more musictherapy sessions. From this sensor-fused data capture, features areextracted and classified to label various key patient therapy responses.Further sensor-fused data analysis uses histograms to allow for initialmusic therapy response pattern detection.

For music therapy session sensor fused data analysis, initially, patientspecific Bayesian inference models are used utilizing Markov chains. Thestates of the chain represent the patient specific response patternscaptured from music therapy baseline sessions. The inference is based onknowledge of the patient response pattern appearances at each sampleinterval and the temporal link to the previous state.

The prediction routine, a Multi-Layer Perceptron Neural Network (MLPNN),uses a directed graph node-based model having a top layer root-nodewhich predicts requirements for reaching a subsequent node and obtaininga patient's sensor-fused data feature vector. This sensor fused datafeature vector contains time-series processed motion data, musicsignature data, and video image data that is specifically significantfor further processing. The directed graph, in this case, look liketrees that are drawn upside down, where the leaves are at the bottom ofthe tree and the roots are the root-node. From each node, the routinecan go to the left, where left is the left node on the next layer belowthe top layer which is where the root-node is located, selecting theleft sub-node as the next observed node, or to the right where right isthe right node on the next layer below the top layer where the root-nodeis located, and this based on the value of a certain variable whoseindex is stored in the observed node. If the value is less than thethreshold, the routine goes to the left node and if greater, it goes tothe right node. These regions, here, left & right, become the predictorspaces.

The model uses two types of input variables: ordered variables andcategorical variables. An ordered variable is a value that is comparedwith a threshold that is also stored in a node. A categorical variableis a discrete value that is tested to see whether it belongs to acertain limited subset of values and stored in a node. This can beapplied to various classifications. For example, mild, medium, andsevere can be used to describe tremor and is an example of a categoricalvariable. Conversely, a fine grained range of values or a numericalscale, can be used to similarly describe tremor but in a numericalfashion.

If the categorical variable belongs to the limited set of values, theroutine goes to the left node and if not, it goes to the right node. Ineach node, a pair of entities: variable_index, decision_rule(threshold/subset) are used to make this decision. This pair is called asplit which splits on the variable: variable_index.

Once a node is reached, the value assigned to this node is used as theoutput of the prediction routine. The Multi-Layer Perceptron NeuralNetwork is built recursively, starting from the root node. All trainingdata, feature vectors, and responses, are used to split the root node,as described earlier; where the entities: variable_index, decision_rule(threshold/subset) segments the prediction regions. In each node theoptimum decision rule on the best primary split is found based on gini“purity” criteria for classification and sum of squared errors forregression. The gini index is based on the measure of total varianceacross a set classes. The gini “purity” criteria referrers to a smallgini index value, indicating that a node contains predominantlyobservations from a single class, which is the desired state.

Once the Multi-Layer Perceptron Neural Network is built, it may bepruned using a cross-validation routine. To avoid model over-fitting,some of the branches of the tree are cut off. This routine may beapplied to standalone decisions. One salient property of the decisionalgorithm (MLPNN), described above, is an ability to compute therelative decisive power and importance of each variable.

The variable importance rating is used to determine the most frequentinteraction type for a patient interaction feature vector. The patternrecognition starts with the definition of a decision space suitable todiscriminate different categories of music therapy responses and musictherapy events. A decision space can be represented by a graph with Ndimensions, where N is the number of attributes or measurementsconsidered to represent the music therapy responses and music therapyevents. The N attributes compose a feature vector or signature which canbe plotted in the graph. After sufficient samples have been inputted,the decision space reveals clusters of music therapy responses and musictherapy events belonging to different categories which is used toassociate new vectors to these clusters.

The dynamic closed-loop rehabilitation platform music therapy systemutilizes several deep learning neural networks for learning andrecalling patterns. In one embodiment, a non-linear decision space isbuilt using the adaptive Radial Basis Function (RBF) model generator.New vectors can be calculated using the RBF model and/or with aK-Nearest Neighbor classifier. FIG. 6 illustrates the workflow of themachine learning sub-system of the dynamic closed-loop rehabilitationplatform music therapy system.

FIG. 7 illustrates the supervised training process 408, which includes anumber of training samples 502, e.g., inputs would be features such asdescribed in Table 3, above and example outputs will be items such astremor, asymmetry, and power, the degree of these items, the predictionof changes, classification of how well the patient is recovering. It isunderstood new outputs are learned as a part of this process. Thisprovides a base for higher levels of abstractions of the predictions andclassifications as it is applied to different use cases (e.g. differentdisease states, combinations with pharmaceuticals, notifications toproviders, fitness, and fall prevention). These training samples 502 arerun with learning algorithms A1 504 a, A2 504 b, A3 504 c . . . AN 504 nto derive predictive models in M1 506 a, M2 506 b, M3 506 c . . . MN 506n. Exemplary algorithms include Multi-Layer Perceptron Neural Networks,Hidden Markov Models, Radal Based Function Networks, Bayesian inferencemodels.

FIG. 8 illustrates the ensemble machine learning system 410, as anaggregation of the predictive models M1 506 a, M2 506 b, M3 506 c . . .MN 506 n on sample data 602 e.g., feature extracted data, to providemultiple predictive outcome data 606 a, 606 b, 606 b . . . 606 n. Anaggregation layer 608, e.g., including decision rules and voting, isused to derive the output 610, given a plurality of predictive models.

The MR ConvNet system has two layers, where the first layer is aconvolutional layer with mean pooling support. The MR ConvNet systemsecond layer is a fully connected layer that supports multinomiallogistic regression. Multinomial logistic regression, also calledSoftmax, is a generalization of logistic regression for handlingmultiple classes. In the case of logistic regression, the labels arebinary.

Softmax is a model that is used to predict the probabilities of thedifferent possible outputs. The following assumes a multiclassclassifier with m discrete classes via a Softmax final output layer:

Y1=Softmax (W11*X1+W12*X2+W13*X3+B1)   [1]

Y2=Softmax (W21*X1+W22*X2+W23*X3+B2)   [2]

Y3=Softmax (W31*X1+W32*X2+W33*X3+B3)   [3]

Ym=Softmax (Wm1*X1+Wm2*X2+Wm3*X3+Bm)   [4]

In general: Y=softmax (W*X+B)   [5]

Softmax (X)i= exp (Xi)/Sum of exp (Xj) from j=1 thru N   [6]

Where Y=Classifier output; X=Sample input (all scaled (normalized)feature values); W=Weight Matrix. The classifications will, for example,score asymmetry, such as “Moderate Asymmetry score 6 out of 10 (10 highlevel of asymmetry to 0 for no asymmetry)” or gait fluidity “GaitFluidity score 8 out of 10 Normal”, etc. The Analytics pipelines isillustrated in FIG. 9.

Softmax regression allows for handling multiple classes beyond two. Forlogistic regression: P(x)=1/(1+exp (−Wx)) where W contains the modelparameters that were trained to minimize a cost function. Also, x is theinput features vector and

((x(1), y(1)), . . . ,(x(i), y(i)))   [7]

would represent the training set. For multi-class classification,Softmax regression is used where y can take on N different valuesrepresenting the classes instead of 1 and 0 in the binary case. So forthe training set ((x(1), y(1)), . . . ,(x(i), y(i))), y(n) can be anyvalue in the range of 1 through N classes.

Next, p(y=N|x;W) is the probability for each value of i=1, . . . , N.The following mathematically illustrates the Softmax regression process:

Y(x)=(p(y=1|x;W), p(y=2|x;W), . . . p(y=N|x;W))   [8]

Where Y(x) is the answer to the hypothesis, that given the input x,output the probability distribution across all classes such that theirnormalized sum is 1.

The MR ConvNet system convolves every windowed biomechanical dataframes, as a vector, with every biomechanical template filter, as avector, and then generates the responses using a mean pool functionwhich averages the feature responses. The convolution process computesWx while adding any biases and then passes this to a logistic regression(sigmoid) function.

Next, in the MR ConvNet system's second layer, the sub-sampledbiomechanical template filter responses are moved into a two dimensionalmatrix where each column represents the windowed biomechanical dataframes as a vector. The Softmax regression activation process is nowinitiated using:

Y(x)=(1/(exp (Wx)+exp (Wx)+ . . . . +exp (Wx))*(exp (Wx), exp (Wx), . .. , (exp (Wx))   [9]

The MR ConvNet system is trained with an optimization algorithm,gradient descent where a cost function J(W) is define and will beminimized:

J(W)=1/j*((H(t(j=1), p(y=1|x;W)+H(t(j=2), p(y=2|x;W)+ . . . +H(t(j),p(y=N|x;W))   10]

Where t(j) are the target classes. This averages all cross-entropiesover the j training samples. The cross-entropy function is:

H(t(j), p(y=N|x;W)=−t(j=1)*log (p(y=1|x;W))+t(j=2)*log (p(y=2|x;W))+ . .. +t(j)*p(y=N|x;W)   [11]

In FIG. 10, the ensemble machine learning system 408 includes aplurality of predictive models, e.g., Template Series 1 (tremor) 706 a,Template Series 2, (symmetry) 706 b, Template Series 3 (fluidity) 706 c. . . additional templates (other learned biomechanical parameters,e.g., entrainment, initiation, etc.) 706 n which are applied toconditioned inputs 702, e.g., for example, it could be the following:stride length for right and left features (x1, x2), variance of stridelength right and left features (x3, x4), cadence right and left features(x6, x7), variance of cadence right and left features (x8, x9) etc . . .this is where sample (x1,x2, . . . xn) are referred to as the Vector Xwhich is input to 702 in the ensemble of ML algorithms. These areconditioned referencing normalized and/or scaled inputs]. Theaggregation classifier 708 outputs such information as tremor scale,symmetry scale, fluidity scale, etc.

Music Therapy Center

The music therapy center 110 is the decision making system that runs onprocessor, such as handheld device or laptop computer 220 of FIG. 2. Themusic therapy center 110 takes the inputs from the feature-in extractedsensor data at the collector 106, compares them to the defined processfor the delivering of the therapy, and then delivers content of auditorystimuli that is played through music delivery system 230.

Embodiments of the invention use contextual information to determine whya situation is happening, then encodes observed actions, which causes adynamic and modulated change in the system-state, and thus the musictherapy session, in a closed-loop manner.

The interactions between the patient and music therapy session providereal-time data for determining music therapy patient context awareness,including motion, posture, strides, and gait reaction. After input datais collected by the sensing nodes (at the sensors), embedded nodesprocess the context-aware data (at edge processing), and provideimmediate dynamic action and/or transmit the data to the analyticssystems 108, e.g., an elastic network-based processing cloud environmentfor storage and further processing and analysis.

Based on inputs, the program will take any existing song content, alterthe cadence, major/minor chords, meter and musical cues (e.g., melodic,harmonic, rhythmic and force cues). The system can overlay a metronomeon existing songs. The song content can be beat mapped (e.g., if W inresponse to AV or MP3 file) or in MIDI format so that the preciseknowledge of when the beat occurs can be used to calculate theentrainment potential. The sensors on the patient can be configured toprovide haptic/vibration feedback pulsing at the music content.

EXAMPLES

An exemplary application of the method is described herein. Gaittraining analyzes the real-time relationship between the beats of themusic being played for the patient and the individual steps taken by thepatient in response to those particular beats of music. As discussedabove, gating analysis is used to determine a window of data thatrepeats, with some variation, with each step or repetitive movement. Insome embodiments, the beginning of the window is determined as the timewhen the heel strike pressure exceeds a threshold (or other sensorparameter.) FIG. 11 is an exemplary time plot illustrating the beats ofmusic, “time beats,” and the steps taken by the patient, “time step.”Thus the onset time in this case is associated with the “time step.” Inparticular, the plot illustrates a time beat 1101 of the music at timeTime Beat 1. After a duration of time, the patient takes a step inresponse to time beat 1001, i.e., time step 1102, at time Time Step 1.The entrainment potential 1103 represents the delay (if any) betweenTime Beat 1 and Time Step 1.

FIGS. 12-13 illustrate examples of entrainment of a patient's gait byuse of the system described herein. FIG. 12 illustrates a “perfect”entrainment, e.g., a constant entrainment potential of zero. This occurswhen there is no delay, or negligible delay, between the time beat andthe associated time step taken in response to the time beat. FIG. 13illustrates a phase-shift entrainment, e.g., a condition in which theentrainment potential is non-zero, but remains constant, or with minimalvariation, over time. This occurs when there is a consistent delay,within tolerances, between the time beat and the time step over time.

With continued reference to FIG. 11, an EP Ratio is calculated as aratio of the time duration between time beats to the time durationbetween time steps:

$\begin{matrix}{{{EP}\mspace{14mu}{Ratio}} = \frac{{{Time}\mspace{14mu}{Beat}\mspace{14mu} 2} - {{Time}\mspace{14mu}{Beat}\mspace{14mu} 1}}{{{Time}\mspace{14mu}{Step}\mspace{14mu} 2} - {{Time}\mspace{14mu}{Step}\mspace{14mu} 1}}} & \lbrack 6\rbrack\end{matrix}$

Where Time Beat 1 1101 corresponds to the time of a first music beat,and Time Step 1 1102 corresponds to the time of the patient's step inresponse to Time Beat 1. Time Beat 2 1106 corresponds to the time of asecond music beat, and Time Step 2 1108 corresponds to the time of thepatient's step in response to Time Beat 2. The goal is for an EP Ratio=1or EP Ratio/Factor=1. The Factor is determined as follows:

$\begin{matrix}{2^{{round}{({{log2}{(\frac{{{Time}\mspace{14mu}{Step}\mspace{14mu} 2} - {{Time}\mspace{14mu}{Step}\mspace{14mu} 1}}{{{Time}\mspace{14mu}{Beat}\mspace{14mu} 2} - {{Time}\mspace{14mu}{Beat}\mspace{14mu} 1}})}})}} = {Factor}} & \lbrack 7\rbrack\end{matrix}$

This factor allows the subdivision of beats to happen or for someone tostep every 3 beats or 3 out of every 4. It can provide flexibility fordifferent scenarios

FIGS. 14 and 15 illustrate the entrainment response over time of apatient using techniques described herein. FIG. 14 (Left Y-axis: EPRatio; Right Y-axis: Beats Per Minute; X-axis: time) illustrates ascattering of dots 1402 which represent the averages of the EP Ratio ofa first patient's gait. The graph illustrates an upper limit 1404 of+0.1 and a lower limit 1406 of −0.1. The lines 1408 illustrate the tempoover time (starting at 60 beats per minute), increasing in steps to 100bpm). FIG. 14 illustrates that the EP Ratio remains near 1 (±0.1) as thetempo is increased from 60 bpm to 100 bpm. FIG. 15 illustrates the EPratio of a second patient's gait, in which the EP Ratio also remainsnear 1 (±0.1) as the tempo is increased from 60 bpm to 100 bpm.

FIGS. 16 and 17 (Y-axis: Entrainment potential, X-axis: Time) illustratetwo patients' responses to a change in the time beats (e.g., change intempo) and/or change to the chords, change in haptic feedback, change incueing of the feet (e.g., left-right, or left-right-cane cueing), etc.FIG. 16 shows a time based plot in which the patient's gait equilibrateswith “perfect entrainment” (constant zero or negligible entrainmentpotential), or a constant phase-shifted entrainment potential. Asillustrated in the figure, it takes a certain period of time, prime time1602, until equilibration occurs. FIG. 17 illustrates a time-based plotin which the patient's gait does not equilibrate, e.g., does not reachperfect entrainment or a constant phase-shifted entrainment potentialafter a change to the time beats. Prime time is useful because itrepresents a set of data that is separate from measuring the accuracy ofentrainment. The prime time parameter can also be used to screen futuresongs for suitability. For example, when patients exhibit a longer primetime value when a music piece is used, such music piece is less capablefor therapy.

FIG. 18 illustrates a technique useful for gait training, wherein therepetitive movement refers to the steps taken by the patient whilewalking. Gait training is adapted to individual patient populations,diagnosis, and conditions to deliver personalized and individualizedmusic interventions. Based on the inputs, the program changes thecontent, cadence, major/minor chords, meter, and musical cues (e.g.,melodic, harmonic, and force cues) where applicable. The program canmake selections of music by using date of birth, listed musicpreferences, and entraining tempo to provide a playlist of passive musicto use on a regular basis. The key inputs for gait training are cadence,symmetry and stride length of the user executing the physical activity,e.g., walking. The program uses connected hardware to providehaptic/vibration feedback at the BPM of the music. The appropriatepopulations for gait training include patients with traumatic braininjury (TBI), stroke, Parkinson's, MS and aging.

The method starts at step 1802. At step 1804, biomechanical data isreceived at the collector 106 based on data from sensors, e.g., sensors200, 206, 208. Biomechanical data includes initiation, stride length,cadence, symmetry, data about assistive device, or other such patientfeature sets that were stored and generated by the analytics systems108. Exemplary biomechanical data parameters are listed in Table 1, 2,and 3 above. The baseline condition is determined from a one or moresources of data. First, the patient's gait without any music beingplayed is sensed. Sensor and feature data regarding the patient'sinitiation, stride length, cadence, symmetry, data about assistivedevice, etc. comprise the patient's baseline biomechanical data for atherapy session. Second, sensor data from previous sessions of the samepatient, as well as any higher level classification data from analyticssystem 108 comprise the patient's historical data. Third, sensor dataand higher level classification data for other similarly-situatedpatients comprise population data. Thus, the Baseline condition caninclude data from one or more of (a) the patient's baselinebiomechanical data for a therapy session, (b) data from the patient'sprevious sessions, and (c) population data. The baseline beat tempo isthen selected from the baseline condition. For example, a baseline beattempo can be selected to match the current cadence of the patient priorto playing music. Alternatively, the baseline beat tempo can be selectedas a fraction or multiple of the current cadence of the patient. Asanother alternative, the baseline tempo can be selected to match thebaseline beat tempo used in the same patient's previous session. As yetanother alternative, the baseline beat tempo can be selected based onbaseline beat tempos used for other patients with similar physicalconditions. Finally, the baseline beat tempo can be selected based on acombination of any of the data described above. A goal beat tempo canalso be determined from this data. For example, the goal beat tempo maybe selected as a percentage increase in the baseline beat tempo byreference to the improvement exhibited by other similarly situatedpatients. The tempo is understood to refer to the frequency of beats inthe music.

At step 1806, music provided to the patient on music delivery device 230(e.g., earbuds or headphones, or a speaker) from handheld device 220 isstarted at baseline tempo or a subdivision of the baseline tempo. Inorder to supply music to the patient at the baseline tempo, music ishaving a constant baseline tempo is selected from a database, orexisting music is modified, e.g., selectively sped up or slow down, inorder to provide beat signals at a constant tempo.

At step 1808, the patient is instructed to listen to the beat of themusic. At step 1810, the patient is instructed to walk at the baselinebeat tempo, optionally receiving cues as to left and right feet. Thepatient is instructed to walk such that each step closely matches thebeat of the music, e.g., to walk “in time” with the beat tempo. Steps1806, 1808, and 1810 may be initiated by the therapist, or by audible orvisual instructions on the handheld device 220.

At step 1812, the sensors 200, 206, 208 on the patient are used torecord patient data, such as heel strike pressure, 6-Dimensionalmovement, EMG activity, and a video record of patient movement. Allsensor data is time-stamped. Data analysis is performed on thetime-stamped sensor data including “gate” analysis discussed herein. Forexample, analysis of the sensor data, e.g., heel strike pressure, ismade in order to determine the onset time of each step. Additional datareceived includes the time associated with each beat signal of the musicprovided to the patient.

At step 1814, a connection is made to the entrainment model (e.g., theensemble machine learning system 410 of the analytics system 108 ormodels downloaded on collector 106 and running on the handheld device220) for prediction and classification. (It is understood that suchconnection may be pre-existing or initiated at this time.) Suchconnection is typically very fast or instantaneous.

At step 1816, an optional entrainment analysis performed at theanalytics systems 108 is applied to the sensor data. The entrainmentanalysis includes the determination of the delay between the beat signaland the onset of each step taken by the patient. As an output from theentrainment analysis, a determination is made regarding the accuracy ofthe entrainment, e.g., a measure of the instantaneous relationshipbetween the baseline tempo and the patient's step as discussed aboveregarding the entrainment potential and EP ratio. If the entrainment isnot accurate, e.g., entrainment potential is not constant within atolerance, adjustments are made at step 1818, e.g., speed up or slowdown the beat tempo, increase volume, increase sensory input, overlaymetronome or other related sound, etc. If the entrainment is accurate,e.g., entrainment potential is constant within a tolerance, anincremental change is made to the tempo at step 1820. For example, thebaseline tempo of the music played with handheld device is increasedtowards a goal tempo, e.g., by 5%.

At step 1822, a connection is made to the entrainment model forprediction and classification. (It is understood that such connectionmay be pre-existing or initiated at this time.) At step 1824, anoptional symmetry analysis is applied to the sensor data. As an outputfrom the symmetry analysis, a determination is made regarding thesymmetry of the patient's gait, e.g., how closely the patient's leftfoot motion matches the patient's right foot motion for stride length,speed, stance phase, swing phase, etc. If the steps are not symmetrical,e.g., below a threshold, adjustments are made at step 1826 to the musicbroadcast to the patient by the handheld device. A first modificationmay be made to the music played during movement of one of the patient'sfeet, and the second modification may be made to music played duringmovement of the other one of the patient's feet. For example, a minorchord (or increased volume, sensory input, change in tempo, or overlayof sound/metronome) may be played on one side, e.g., an affected side,and a major chord played on the other side, e.g., a non-affected side.The machine learning system 410 predict in advance when symmetryproblems are coming based on the ‘fingerprint’ of the scenarios leadingup to it, e.g., by analyzing motions that are indicative of asymmetry.Asymmetry can be determined by comparing the normal gait parameters forsomeone with their background can determine how affected the side is andcompared to other side.

At step 1828, connection is made to the entrainment model for predictionand classification. (It is understood that such connection may bepre-existing or initiated at this time.) At step 1830, an optionalcenter of balance analysis, e.g., whether the patient is leaningforward, is performed on the sensor data. The analysis may be performedby combining outputs of the foot sensors, as well as the video output.As an output from the center of balance analysis, a determination ismade regarding whether the patient is leaning forward. If the patient isleaning forward, a cue to the patient to “stand up straight” is made atstep 1832, provided by the therapist, or by audible or visualinstructions on the handheld device.

At step 1834, a connection is made to the entrainment model forprediction and classification. (It is understood that such connectionmay be pre-existing or initiated at this time.) At step 1836, aninitiation analysis is applied to the sensor data, e.g., patient'sexhibits hesitation or difficulty initiating walking. As an output fromthe initiation analysis, a determination is made regarding the whetherthe patient exhibits a problem with initiation. If the patient exhibitsa problem with initiation, e.g., below a threshold, haptic feedback canbe provided to the patient, which may include a countdown at the beattempo or a countdown prior to the beginning of a song at step 1838.

At step 1840, it is optionally determined whether the patient is usingan assistive device, e.g., a cane, crutches, walker, etc. In someembodiments, the handheld device 220 provides a user interface for thepatient or therapist to enter information regarding the use of anassistive device. If a cane is present, the analysis is changed to threemeter, e.g., cane, right foot, left foot, and cueing by “left foot,”“right foot,” and “cane,” is made at step 1842, provided by thetherapist, or by audible or visual instructions on the handheld device.

At step 1844, a connection is made to the entrainment model forprediction and classification. (It is understood that such connectionmay be pre-existing or initiated at this time.) An optional entrainmentanalysis 1846 is applied to the sensor data, substantially as describedabove in step 1816, with the differences noted herein. For example,entrainment may be compared with previous entrainment data from earlierin the session, from previous sessions with the patient, or with datarelating to entrainment of other patients. As an output from theentrainment analysis, a determination is made regarding the accuracy ofthe entrainment, e.g., how closely the patient's gait matches thebaseline tempo. If the entrainment is not accurate, adjustments are madeat step 1848, substantially in the same manner as described above atstep 1818.

If the entrainment is accurate, a determination is made at step 1850whether the patient is walking at the goal tempo. If the goal tempo isnot reached, the method proceeds to step 1820 (described above), so thatan incremental change is made to the tempo. For example, the baselinetempo of the music played with handheld device is increased ordecreased, e.g., by 5%, towards the goal tempo. If the goal tempo hasbeen reached, the patient may continue the therapy for the remainingtime in the session (step 1852). At step 1854, music at the desiredtempo to be used when not in therapy session can be curated and left onthe device 220 in FIG. 2. This music content is used ashomework/practice by the patient between dedicated therapy sessions. Atstep 827, the program ends.

Is understood that the steps described above and illustrated in FIG. 18may be performed in a different order than that disclosed. For example,the evaluations at steps 1816, 1824, 1830, 1836, 1840, 1846, and 1850may be performed at the same time. Moreover, the plurality ofconnections to the analytics system 108 (e.g., steps 1814, 1822, 1828,1834, and 1844) may be performed once throughout the therapy sessiondescribed.

FIG. 19 illustrates a technique useful for neglect training. For neglecttraining, the system and methods described herein use connected hardwareto provide haptic/vibration feedback as the patients correctly hit thetarget. The connected hardware includes a device, video motion capturesystem or connected bell. All of these devices connect into the systemdescribed, vibrate as tapped, and have a speaker to play auditoryfeedback. For example, the connected bell provides data to the system inthe same manner as the sensors 200, e.g., data regarding the bell strikeby the patient. The video motion capture system provides video data tothe system in the same manner as the video cameras 206. The key inputsfor neglect training are information relating to the tracking ofmovement to a specific location. The program uses connected hardware toprovide haptic/vibration feedback as the patient correctly hits thetarget. The appropriate populations for neglect training includepatients with spatial neglect or unilateral visual neglect conditions.

The flow diagram illustrated in FIG. 19 for neglect training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For example, a baselinetest establishes the status of the patient and/or improvement fromprevious tests. In some embodiments the baseline tests include showingfour objects evenly spaced left to right on a screen, e.g., display 222of handheld device 220. The patient is instructed, either by cuesappearing on the display 222 or verbally by a therapist, to strike theobject in time with the beats of the background music. As with gaittraining, the patient is instructed to strike a bell in time with thebeat of the background music. Every accurate strike provides feedback.Once the baseline information is collected, a number of objects evenlyspaced left to right are displayed on a screen. As above, the patient isinstructed to strike the objects in order from left to right in timewith the beats of the background music. Every accurate strike provides afeedback. As with gait training, the analytics system 108 evaluates thepatient's responses and classifies the responses and provideinstructions to add or reduce objects, or increase or decrease tempo ofthe music to reach a goal tempo

FIG. 20 illustrates a technique useful for intonation training. Forintonation training, the system and methods described herein relies onvoice processing algorithms. The phrases typically chosen are commonwords in the following categories: bilabials, gutturals, and vowels. Thehardware is connected to a patient to provide haptic feedback at thebeats per minute to one hand of the patient. The key inputs forintonation training are the tone of voice and words spoken and rhythm ofspeech. The appropriate populations for intonation training includepatients with Broca's aphasia, expressive aphasia, non-fluent aphasia,apraxia, autism spectrum disorder, and Down's syndrome.

The flow diagram illustrated in FIG. 20 for intonation training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For example, hapticfeedback is provided to one hand of the patient to encourage tapping.The patient is then instructed, either by cues appearing on the display222 or verbally by a therapist, to listen to the music played. Thespoken phrase to be learned is played by separating it into two parts,with the first one of the two parts being high-pitched and the second ofthe two parts being low pitched. The patient is then instructed, eitherby cues appearing on the display 222 or verbally by a therapist, tosinging the phrase with the device using the two pitches being played.As with gait training, the analytics system 108 evaluates the patient'sresponses and classifies the responses in terms of accuracy of pitch,words spoken, and ranking by patient or assistant/therapist, and provideinstructions to provide alternate phrases and compare responses totargeted speech parameters.

FIG. 21 illustrates a technique useful for musical stimulation training.For musical stimulation training, the system and methods describedherein relies on voice processing algorithms. Familiar songs are usedwith an algorithm to separate the anticipatory section out (referred toas an expectancy violation). The hardware includes a speaker forreceiving and processing the singing by the patient, and in someembodiments a therapist can manually provide an input regarding singingaccuracy. Key inputs are information relating to the tone of voice andwords spoken and rhythm of speech, and music preferences. Theappropriate populations include patients with Broca's aphasia,non-fluent aphasia, TBI, stroke, and primary progressive aphasia.

The flow diagram illustrated in FIG. 21 for musical stimulation trainingis substantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For example, a song isplayed for the patient, and the patient instructed either by cuesappearing on the display 222 or verbally by a therapist, to listen tothe song. Musical cues are added to the song. Subsequently, atanticipatory spots, a word or sound is left out and gestural music cuesare played to prompt the patient to sing the missing word or sound. Aswith gait training, the analytics system 108 evaluates the patient'sresponses and classifies the responses in terms of accuracy of pitch,words spoken, and ranking by patient or assistant/therapist, and provideinstructions to play additional portions of the song in order to improvespeech to targeted speech parameters.

FIG. 22 illustrates a technique useful for gross motor training. Forgross motor training, the system and methods described herein aredirected to help with ataxia, range of motion or initiation. The morechallenging portion of an exercise is musically “accented”, e.g., by theuse of melodic, harmonic, rhythmic, and/or force cues. Key inputs areinformation relating to movements in X, Y, and Z-capture via connectedhardware or video camera system. The appropriate populations includepatients with neurological, orthopedic, strength, endurance, balance,posture, range of motion, TBI, SCI, stroke, and Cerebral Palsy.

The flow diagram illustrated in FIG. 22 for gross motor training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. As with gait training, thepatient's is provided with cues to move in time with the baseline beatsof a musical selection. The analytics system 108 evaluates the patient'sresponses and classifies the responses in terms of accuracy of motionand entrainment as discussed above and provides instructions to increaseor decrease the tempo of the music played.

FIG. 23 illustrates a technique useful for grip strength training. Forgrip strength training, the system and methods described herein rely onsensors associated with the gripper device. The hardware includes agripper device having pressure sensors, a connected speaker associatedwith a handheld device 220. Key inputs are the pressure provided by thepatient to the gripping device in a similar manner to the heel strikepressure measured by sensor 200. The appropriate populations includepatients with neurological, orthopedic, strength, endurance, balance,posture, range of motion, TBI, SCI, stroke, and Cerebral Palsy.

The flow diagram illustrated in FIG. 23, for grip strength training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. As with gait training, thepatient is provided with cues to apply force to the gripping device intime with the baseline beats of a musical selection. The analyticssystem 108 evaluates the patient's responses and classifies theresponses in terms of accuracy of motion and entrainment as discussedabove and provides instructions to increase or decrease the tempo of themusic played.

FIG. 24 illustrates a technique useful for speech cueing training. Forspeech cueing training, the system and methods described herein relieson voice processing algorithms. The hardware can include a speaker forreceiving and processing the singing by the patient, and in someembodiments a therapist can manually provide an input regarding speechaccuracy. Key inputs are the tone of voice and words spoken and rhythmof speech, and music preferences. The appropriate populations includepatients with robot, word finding and stuttering speech issues.

The flow diagram illustrated in FIG. 24 for speech cueing training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. As with gait training, thepatient is provided with cues to speak a sentence, either by cuesappearing on the display 222 or verbally by a therapist, by saying onesyllable in time with each beat of a musical selection. The analyticssystem 108 evaluates the patient's speech and classifies the responsesin terms of accuracy of speech and entrainment as discussed above andprovides instructions to increase or decrease the tempo of the musicplayed.

FIG. 25 illustrates a technique useful for training of minimallyconscious patients. The system and methods described herein rely on animaging system, such as a 3-D camera, to measure if the eyes of thepatient are open, the direction the patient is looking, and theresulting patient pulse or heart rate. The program searches andoptimizes for the heart rate, stimulation, respiration rate, eyeclosure, posturing, and restlessness. The appropriate populationsinclude patients with coma and disorders of consciousness.

The flow diagram illustrated in FIG. 25 for training of minimallyconscious patients is substantially identical to the flow illustrated inFIG. 18 for gait training, with the differences noted herein. As withgait training, the patient is provided with increasing stimulation atthe breathing rate of the patient (PBR). For example, the patient isfirst provided with stimulation at the PBR of musical chords andobserving whether the patient's eyes are open. If the patient's eyes arenot open, the stimulation sequentially increases from humming a simplemelody at PBR, to singing “aah” at the PBR, to singing the patient'sname at the PBR (or playing a recording of such sounds), and checking ateach input whether the patient's eyes are open. The analytics system 108evaluates the patient's eye tracking and classifies the responses interms of level of consciousness and provides instructions to change thestimulation.

FIGS. 26-28 illustrate a technique useful for attention training. Forattention training, the system and methods described herein operate in aclosed loop fashion to help patients sustain, divide, alternate andselect attention. No visual cue is allowed to signal which movements tomake. The appropriate populations include patients with brain tumor,multiple sclerosis, Parkinson's disease, and neurological disease andinjury.

The flow diagram illustrated in FIG. 26 for sustained attention trainingis substantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. As with gait training, thepatient is provided with an instrument (e.g., any instrument could work,such as a drumstick, drum, keyboard, or wireles sly connected version ofeach) and is instructed, either by cues appearing on the display 222 orverbally by a therapist, to follow along or perform a task to audio cuesdefined by levels 1 through 9 as illustrated in FIG. 26. The analyticssystem 108 evaluates the patient's ability to accurately complete thetask and classifies the responses to change the tempo or the difficultyof the task. Similarly, FIG. 27 illustrates a flow diagram foralternating attention training in which the instructions are provided,either by cues appearing on the display 222 or verbally by a therapist,to follow along or perform a task to audio cues which alternate betweenthe left and the right ear. FIG. 28 illustrates a flow diagram fordivided attention in which the instructions are provided to follow alongor perform a task to audio cues with audio signals in both the left andright ear.

The flow diagram illustrated in FIG. 29 for dexterity training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For dexterity training, thepatient is instructed to tap with their fingers on the keyboard of thepiano to gather baseline movement and range of motion information. Thesong is started at a particular beat per minute, and the patient startstapping with the baseline number of fingers. The analytics system 108evaluates the patient's ability to accurately complete the task andclassifies the responses to change the tempo or the difficulty of thetask.

The flow diagram illustrated in FIG. 30 for oral motor training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For oral motor training,the patient is instructed to perform a task alternating between twosounds, e.g., “ooh” and “aah.” The analytics system 108 evaluates thepatient's ability to accurately complete the task and classifies theresponses to change the tempo or the difficulty of the task, e.g., byproviding a different target sound.

The flow diagram illustrated in FIG. 31 for respiratory training issubstantially identical to the flow illustrated in FIG. 18 for gaittraining, with the differences noted herein. For respiratory training, abaseline breathing rate and shallowness of breathing is determined.Music is provided with a baseline tempo at the patient's breathing rate,and the patient is instructed to perform breathing tasks has describedin the levels in FIG. 31. The analytics system 108 evaluates thepatient's ability to accurately complete the task and classifies theresponses to change the tempo or the difficulty of the task, e.g., byproviding a different breathing pattern

Further described herein are methods, system, and apparatus for usingaugmented reality (AR) and augmented audio (AA) to support the nextgeneration of medical and therapy systems for the improvement ormaintenance of motor functions. Exemplary embodiments of the augmentedneurologic rehabilitation, recovery or maintenance (“ANR”) systems andmethods disclosed herein build upon entrainment techniques by utilizingadditional sensor streams to make determinations of therapeutic benefitand inform a closed loop therapeutic algorithm. Exemplary systems andmethods for neurologic rehabilitation, which can be utilized to realizeembodiments of the ANR systems and methods, are shown and describedabove and in co-pending and commonly assigned U.S. patent applicationSer. No. 16/569,388 for “Systems and Methods for NeurologicRehabilitation,” to McCarthy et al., which is a continuation of U.S.Pat. No. 10,448,888, titled, “Systems and Methods for NeurologicRehabilitation,” issue date Oct. 22, 2019, which is based on and claimspriority to U.S. Provisional Patent Application No. 62/322,504 filed onApr. 14, 2016, entitled “Systems and Methods for NeurologicRehabilitation,” which are each hereby incorporated by reference hereinas if set forth in their respective entireties herein.

In accordance with one or more embodiments, the ANR systems and methodscan include a method for providing an AR 3-D dynamic model of specificpeople or objects that include obtaining the images, videos of thepeople or images by querying cloud and local databases. The ANR systemsand methods can also be configured to fuse and/or synchronize thedynamic models, patients or humans, audio content, and context about theenvironment into a synchronized state taking advantage of theneuroscience of the ability of music to improve motor function as wellas the neuroscience of how visual imagery can impact recovery (e.g. theuse of mirror neurons).

In accordance with one or more embodiments, the ANR systems and methodsinclude a method for combining AA techniques for repetitive motionactivities. Augmented Audio (AA) combines real world sound withadditional computer-generated audio “layers” that enhance sensory input.The neuroscience of rhythm at its core uses the stimulus to engage themotor system for repetitive motion activities such as walking. Adding AAto the therapy protocols enhances therapeutic effect, increasesadherence to the therapy protocols and provides greater safety in theform of enhanced situational awareness to the patient. The disclosedembodiments configured for adding AA can mix many audio signals,including external environmental sound inputs, recorded content,rhythmic content, and voice guidance into a synchronized state takingadvantage of the neuroscience of music. Additionally, it can include theability to combine algorithmically generated music with underlyingrhythmic cueing as detailed in [00218] for patients with motor orphysical disabilities. This can be done by fusing this generative rhythmwith inputs from a patient's real-time biometric data into aninteractive feedback state. Exemplary systems and methods foralgorithmically generating auditory stimulus for use in connection withneurologic rehabilitation are shown and described in co-pending andcommonly assigned U.S. patent app. Ser. No. 16/743,946, filed Jan. 15,2020, titled “ENHANCING MUSIC FOR REPETITIVE MOTION ACTIVITIES” toMcCarthy et al., which is a continuation of U.S. patent application Ser.No. 16/044,240 filed Jul. 24, 2018, now U.S. Pat. No. 10,556,087 issuedon Feb. 11, 2020, entitled “ENHANCING MUSIC FOR REPETITIVE MOTIONACTIVITIES”, which claims the benefit of priority under 35 USC 119(e) ofU.S. Provisional Application No. 62/536,264 filed Jul. 24, 2017, theentire contents of each of which are hereby incorporated herein as ifset forth in their respective entireties herein.

Neuroplasticity, entrainment, the science of mirror neurons are thefoundational scientific components supporting the disclosed embodiments.Entrainment is a term for the activation of the motor centers of thebrain in response to an external rhythmic stimulus. Studies have shownthat audio-motor pathways exist in the Reticulospinal tract, the part ofthe brain that is responsible for movement. Priming and timing ofmovements via these pathways demonstrate the motor system's ability tocouple with the auditory system in order to drive movement patterns(Rossignol and Melville, 1976). The entrainment process has been shownto effectively increase walking speed (Cha, 2014), decrease gaitvariability (Wright, 2016), and lower fall risk (Trombetti, 2011).Neuroplasticity refers to the brain's ability to strengthen preexistingneural connections and, thus, allow an individual to acquire new skillsover time. Studies have demonstrated that music facilitates changes incertain motor regions of the brain, indicating that music can promoteneuroplasticity (Moore et al., 2017). Mirror neurons fire both when youperform an action and when you observe another performing an action.Thus, when you see another performing an action, your brain responds asthough you were the one doing it. This allows us to learn behaviors byimitation. This is important in the context of the disclosed embodimentsbecause as patients observe augmented reality human simulations, theycan imitate their actions using mirror neurons.

In accordance with one or more embodiments, the ANR systems and methodsare configured to process the images/videos to remove/add people/objectssmaller or larger than a specified size from the images/videos, inresponse to patient or therapist exception conditions received as inputsto the ANR system. Such exceptions can be a patient response such as aninstruction to reduce scene complexity, a therapist instruction tointroduce occlusions which could be people/objects increasing scenecomplexity. The embodiments can support recording all data of allpatient or therapist exception conditions besides the session dataitself.

In accordance with one or more embodiments, the ANR systems and methodsinclude a telepresence method allowing the linking of a therapist to aremotely located patient using the system. The telepresence method,besides fully supporting all the local features experienced by patientand therapist when being in the same location, includes biomechanicalmotion tracking of the patient relative to the AR 3-D dynamic model ofpeople/objects.

The AR 3-D dynamic model is a software-based algorithmic process of theANR systems and methods for generating a presenting an AR/VR visualscene to the patient that is animated based on the foundationalprincipals of neuroplasticity, entrainment, and the science of mirrorneurons to facilitate outcomes towards a clinical or training goal. Thetelepresence method is configured to provide the ability to operate aninteractive video link between the therapist and remotely locatedpatient using the invention. The telepresence method supports projectingimage/video from the remotely located patient with the invention toindicate the relative position of the patient in the AR 3-D dynamicmodel. The telepresence method can also provide the ability for thetherapist to adjust AR models in real time (spatial location or whichitems are available) and allow modifications to the session. It willalso allow a therapist to see the patient relative to the models.

FIG. 32 depicts a conceptual overview of principal components of anexemplary ANR system 3200 that uses a closed loop feedback thatmeasures, analyzes, and acts on a person to facilitate outcomes towardsa clinical or training goal. It should be understood that the ANR system3200 can be realized using the various hardware and/or softwarecomponents of the system 100 described above. As shown, the ANR systemmeasures or receives inputs relating to gait parameters, environment,context/user intent (including past performance), physiologicalparameters, and real time feedback on outcomes (e.g. closed loop, realtime decision making). One or a combination of these inputs can bedirected into the clinical thinking algorithm module (CTA) 3208, whichis in control of analyzing and acting on this information. In someembodiments, an example of real-time feedback would be a determinationby the CTA 3208 that the user's measured quality of gait metrics (e.g.symmetry, stability and gait cycle time variance) have exceeded modeledsafety thresholds, thus triggering a new cueing response. Such cueingresponses are both visual and audio, for example, increasing the beatsalience in the music by adding a metronome audio layer and modifyingthe AR scenery to model or direct the user into a safer gait speed andmovement behavior.

At the CTA module 3208, the actions performed by the ANR system aredefined based on the analysis of the inputs. The actions can be outputto the patient in the form of various types of stimuli, including music(or other components), rhythm, sonification of sound, spoken words,augmented reality, virtual reality, augmented audio, or tactilefeedback. As shown in FIG. 32, determinations made by the criticalthinking algorithm relating to the various inputs, outcomes etc., areprovided to an AR/AA output modelling module 3210 programmed todynamically generate/modify outputs for the patient. The outputs areprovided to the patient via one or more output devices 3220, such asvisual and/or audio output devices and/or tactile feedback devices. Forexample, AR visual content can be output to AR glasses 3222 worn by thepatient. Augmented audio content can be provided to the patient viaaudio speakers or headphones 3225. As would be understood, othersuitable visual or audio display devices can be used without departingfrom the scope of the disclosed embodiments. Additionally, althoughvarious elements of the system are shown separately in FIG. 32, itshould be understood that features and functionality of various aspectsof the system can be combined.

The clinical thinking algorithm module 3208 in this medical/therapysystem can be configured to implement a critical thinking algorithmfocusing on the recovery, maintenance, or enhancement of motor function,including but not limited to upper extremity, lower extremity,agitation, postural stability, foot drop, dynamic stability, breathing,mouth movements, respiratory, endurance, heart rate training, breathingfrequency, optical flow, boundary support training, strategy training(ankle, knee, and hip), attractor coupling, muscle firing, trainingoptimization, and gait. One or more of these CTA's could also beimplemented in synchrony or combination with other interventions thatshare similar goals. Examples could include implementing CTAs incombination with functional electrical stimulation, deep brainstimulation, transcutaneous electrical nerve stimulation (TENS), Gammafrequency audio entrainment (20-50 Hz) or other electrical stimulationsystems. Additionally, dosing and operations of CTA's could be combinedwith anti-spasticity medications or dosing in combination withneurotoxin injections. For example, the CTA could be applied or startduring the time window that has been shown these interventions are attheir peak effect or to be used prior to them getting to that point as away to prime the motor system.

ANR System Inputs

The inputs to the ANR system 3200 are important to enable the system tomeasure, analyze, and act in a continuous loop facilitating outcomestowards a clinical or training goal.

Receiving an input at the system can include measuring the movements ofthe person via a sensor to determine the biomechanical parameters ofmovements (e.g. temporal, spatial, and left/right comparisons). Thesemotion sensors could be placed anywhere on the body and could be asingle sensor or an array of sensors. Other types of sensors could beused to measure other input parameters, which could include respiratoryrate, heart rate, oxygen level, temperature, electroencephalogram (EEG)for recording of the brain's spontaneous electrical activity,electrocardiogram (ECG or EKG) for measuring the electrical activity ofthe heart, electromyogram (EMG) for evaluating and recording theelectrical activity produced by skeletal muscles, photoplethysmogram(PPG) for detecting blood volume changes in the microvascular bed oftissue often using a pulse oximeter which measures changes in lightabsorption of skin, optical, inertial measurement units, video cameras,microphones, accelerometers, gyroscopes, infrared, ultrasonic, radar, RFmotion detection, GPS, barometers, RFID's, radar, humidity, or othersensors that detect physiological or biomechanical parameters. Forinstance, in the exemplary ANR system 3200 shown in FIG. 32, gaitparameters can be measured using one or more sensors 3252 such as IMUs,footpad sensors, smartphone sensors (e.g., accelerometers) andenvironmental sensors. Additionally, physiology parameters can bemeasured using one or more sensors 3254 such as PPG, EMG/EKG andrespiratory rate sensors.

Additionally, contextual information about the outcomes desired, the useenvironment, data from past sessions, other technologies, and otherenvironmental conditions can be received as inputs to the CTA module andadjust the CTA's response. For instance, as shown in FIG. 32, contextualinformation 3258 and environment input 3256 information can be receivedas inputs that further inform operation of the CTA 3208. An example ofusing contextual information is that information from the past about auser's gait pattern could be used in combination with ArtificialIntelligence (AI) or Machine Learning (ML) systems to provide morepersonalized clinical goals and actions for the patient. These goalscould modify target parameters such as limits on steps per minute,walking velocity, heart rate variability, oxygen consumption (VO2 max),breathing rate, session length, asymmetry, variability, distance walked,or desired heart rate.

Additionally, environmental information could be detected usingBluetooth Low Energy (BLE) beacons, or other wireless proximitytechniques, such as wireless triangulation or angle of arrival, tofacilitate wireless location triggers to have people/objects appearand/or disappear in the patient's field of view with respect to an AR3-D dynamic model depending on detected location. In some embodiments,the AR 3-D dynamic model that is output by the ANR system can becontrolled by the therapist and/or beacon triggers to change or maintainnavigation requirements for the patient. These triggers could be usedwith gait or physiological data as described above to provide additionaltriggers, beside the wireless beacon triggers. For example, gait datafeedback from IMU products allows for a gait feedback loop that providesthe ANR system 3200 with the ability to effect change in the AR 3-Ddynamic model software process.

Clinical Thinking Algorithm

In accordance with one or more embodiments, the CTA module 3208implements clinical thinking algorithms that are configured to controlthe applied therapy to facilitate outcomes towards a clinical ortraining goal. Clinical goals could include items such as thosediscussed in connection with FIGS. 18 through FIGS. 31, and, by way offurther example, interventions for agitation in Alzheimer's, dementia,bipolar disorder, and schizophrenia, and training/physical activitygoals. This section discusses different non-limiting exemplarytechniques that can be used to deliver an appropriate rehabilitationresponse as determined by the CTA, for example, modulating the rhythmictempo and the synchronized AR visual scenery. Each of these techniquescould be implemented using a stand-alone CTA or combined with eachother. In one or more embodiments, the system 3200 can be configured tocombine CTA(s) with entrainment principles for repetitive motionactivities and, in other cases, they can be combined with each othertowards other goals.

By way of example and without limitation, the CTA module 3208 of the ANRsystem 3200 can be configured to utilize the combination of thebiomechanical, physiological data, and context to create a virtualtreadmill output via the AR/AA output interfaces. While a treadmillkeeps pace for someone with the movement of the physical belt, thevirtual treadmill is dynamically adjusted using the CTA in accordancewith the entrainment principle to modulate a person's walking ormovement pace in a free-standing manner, similar to other movementinterventions. Though, in addition to or instead of using a rhythmicstimulus to drive the individual towards a bio-mechanical goal asdiscussed previously, the virtual treadmill can be generated anddynamically controlled based on entrainment of the patient towardstarget parameters such as those listed above as target parameters.

The target parameters could be set based on input from a clinician, auser, historic data, baseline conditions, or towards a clinicallymeaningful point. Additional, target parameters could be set by orfollow recommendations on exercise length, duration, and intensityspecific to certain conditions such as heart disease, asthma, COPD, fallprevention, musculoskeletal conditions, osteoarthritis, and generalaging.

Further described herein is an exemplary embodiment of the ANR system3200 in which the CTA module 3208 is configured to utilize thebiomechanical data, physiological data, and context to provide gaittraining therapy in the form of a virtual treadmill and rhythmicauditory stimulus output via the AR and AA output interfaces 3220.

FIG. 37 is a process flow diagram illustrating an exemplary routine 3750for providing gait training therapy to a patient using the ANR system3200. FIG. 38 is a hybrid system and process diagram conceptuallyillustrating aspects of the ANR system 3200 for implementing thegait-training routine 3750 in accordance with exemplary embodiments ofthe disclosed subject matter. As shown, sensors 3252, particularly footmounted IMUs, capture the sensor data relating to gait parameters thatare provided to the CTA module 3208. Additionally, the AA/AR modellingcomponent 3210 comprising an audio engine receives inputs from the CTAand is configured to generate an audio cueing ensemble comprising one ormore of rhythmic music and cuing content, interactive voice guidance andspatial and audio effects processing. Similarly, the AA/AR modellingcomponent 3210 comprising an AR/VR modelling engine (also referred to asAR 3-D dynamic model) is shown as receiving inputs from the CTA and isconfigured to generate a visual cueing ensemble comprising one or moreof virtual AR actors and objects (e.g., a virtual person walking),background motion animation (e.g., virtual treadmill, steps/footprintsand animations) and scene lighting and shading.

FIG. 39 is a hybrid system and process diagram conceptually illustratingan exemplary audio output device 3225 and augmented audio generationcomponents of the ANR system 3200 in greater detail. As shown in FIG.39, in an embodiment, the AA device can capture environmental soundsusing, for example, stereo microphones. The AA device can also generateaudio outputs using stereo transducers. The AA device can also comprisehead-mounted IMUs. As would be understood the AA device can alsocomprise audio signal processing hardware and software components forreceiving, processing and outputting the augmented audio contentreceived from the AA/AR module 3210 alone or in combination with othercontent such as environmental sounds. As shown, the CTA module 3208receives gait parameters including those received from sensors includingfoot mounted IMUs and head mounted IMUs. Additionally, in an embodimentthe CTA receives data relating to physiological parameters from othersensor devices such as PPG sensors.

Returning now to FIG. 37, at step 3700 a patient wearing the AR/AAoutput device 3220 and IMU sensor 3252 starts to walk as the ANR system3200 calibrates and collects preliminary gait data such as stridelength, speed, gait cycle time and symmetry. At step 3701 the CTA module3208 determines the baseline rhythmic tempo for both the music playbackand virtual AR scene to be displayed. For example, the baseline rhythmictempo can be determined by the CTA as described in connection with FIG.18.

With the user's gait cycle times as an input, at step 3702 the audioengine (i.e., the audio modelling component of AR/AA modelling module3210) generates the corresponding tempo adjusted music along with anysupplementary rhythmic cueing, such as a metronome sound, to reinforcethe beat saliency.

At step 3703, the visual AR engine (i.e., the visual modelling componentof AR/AA modelling module 3210) will generate a moving virtual scene,such as those understood in the video game industry. More specifically,in an embodiment, the virtual scene includes visual elements that arepresented under the control of the CTA and share the common timingreference with the audio engine, in order to synchronize the elements ofthe visual scene to the music and rhythm tempo. Although the AR scenedescribed herein includes a virtual treadmill or virtual person andfootsteps, the AR scene could be any one or more of a variety ofexamples discussed herein, such as a virtual treadmill, a virtual personwalking, a virtual crowd or dynamic virtual scene.

At steps 3704 and 3705, the music/rhythm and visual content aredelivered to the patient using an AR/AA device 3220 such as alightweight heads-up display (e.g., AR goggles 3222) with earphones3225. Under the control of the CTA, the patient receives instructions atsteps 3706 and 3707 regarding the therapy via voiceover cues generatedby the audio engine. This could include a pre-walk training preview inorder for the patient to become accustomed to and practice with thevisual scenery and audio experience.

FIG. 40 shows an exemplary AR virtual scene 4010 presented for thepatient to entrain with. As shown, the scene can comprise an animated3-D image of another person walking “in front” of the patient and whosesteps and walking motions are synchronized to the music tempo. Morespecifically, in an embodiment, the AR actor walks to the same tempo asthe baseline beat tempo of the audio content generated by the CTA andaudio engine. In this example, the patient goal can be to match theirsteps both rhythmically with the audio, and visually with the actor.Additionally, as shown in FIG. 40, the AR scene can comprise a pluralityof footsteps with additional cues such as L and R indicating left andright foot.

The scene including the footsteps can be virtually moving toward thepatient at a prescribed rate, while the virtual actor is walking infront of the patient in the direction away from the patient. In anembodiment, the scene can be moving according to a gait cycle time,wherein GCT (right foot)/2*60=music tempo defined by the CTA. Moreover,additional cues generated in connection with the AR scene can includerhythmic audio cues that reinforce the visual cues. For example, oneeffective reinforcement method can include the AA system 3210 generatingthe sound of the virtual actor's footfalls in synchrony with the rhythm,simulating group-marching to a common beat. By providing the patientwith rhythmic audio components generated based on the visual stimulus inaddition to timing the motion of visual elements with the rhythmic audiostimulus, the system further reinforces the virtuous cycle among thefundamental therapeutic concepts of entrainment and mirror neurons.

By way of further example, FIG. 41 shows an exemplary AR virtual scene4110 presented for the patient to entrain with. As shown in FIG. 41, thevirtual treadmill can be generated and dynamically controlled based onentrainment of the patient towards target parameters such as thoselisted above as target parameters. In this example, the generated ARtreadmill animates movement of the treadmill surface 4115 and generatesvirtual steps at the same tempo as defined by the CTA for the auditorystimulus. Additionally, the 3-D animation of a virtual treadmill couldinclude visually highlighted steps or tiles that a patient can use asvisual goals while simultaneously entraining to the rhythm generatedunder control of the CTA. Accordingly, in this example, the patient goalis to match their steps both rhythmically to the audio and visually withthe animated goal steps. As further shown in FIG. 41, animated footsteps4120 can be shown on the surface of the virtual treadmill 4115 moving inthe direction toward the patient illustrated by the directional arrow,which may be reversed if the patient is performing a backwards-walkingtraining exercise. Additionally, animations highlighting goal steps4125L (left foot step) and/or 4125R (right foot step) are respectivelyhighlighted in sync with CTA's rhythm to prompt the user to step withthe corresponding foot (e.g., left or right). Additionally, as furtherdescribed herein, the virtual scenes such as those shown in FIGS. 40-41,can be dynamically adjusted according to the patient's entrainmentpotential (EP) and corresponding changes to the audio stimulus. Otheradjustments to the virtual scene can include changing the virtualbackground environment to simulate different walking scenarios, weather,surfaces, lighting and inclination.

Returning now to FIG. 37, as the patient starts to walk, at step 3708,the biomechanical sensors (e.g., sensors 3252) measure real-time datafor use in evaluating the patient's entrainment level. At step 3709, theentrainment potential (e.g., as explained in FIG. 18) is determined bythe CTA module 3208 and used to determine how the training session goalis to be met. As discussed in previous embodiments of the disclosure,entrainment potential can be the basis for modifying the rhythmic audiostimulus and visual scenery, which occurs at step 3710. For instance,the CTA analyses the incoming data history of the patient's gait cycletimes in comparison to the rhythmic intervals of the beats delivered tothe patient by the audio device. Exemplary approaches for modifying theaudio stimulus based on entrainment potential are similarly describedabove. In one example, if the EP value calculated for the patient'ssteps over a period of time are not within a prescribed range ofacceptable EP values and/or sufficiently consistent, then the CTA modulecan instruct the AA/AR modelling module to adjust (e.g., reduce) thetempo of the RAS and correspondingly adjust the motion speed of the ARscene in sync with the RAS. In a further example, if enough step timesare in phase with the beat times, then the patient is considered to beentraining by the CTA.

Once entraining, at step 3711, the CTA evaluates whether the patient hasreached a goal. If a goal has not been reached, then one or more targetparameters can be adjusted at step 3712. For instance, in one example,the CTA compares the RAS tempo and associated AR scene speed to a targettempo parameter (e.g., a training/therapy goal) before a rhythmic tempoand/or scenery motion speed is changed in view of the comparison.Exemplary methods that the CTA module 3208 can implement for adjustingthe rhythmic auditory stimulus according to entrainment potential areshown and described above, for example, in connection with FIG. 18.

For instance, if the patient has not reached their training speed goal,modifying the target parameters could include increasing or decreasingthe music tempo at step 3712. This would drive the patient to walkfaster or slower using the RAS mechanism of action. Alternatively,another training goal could be to lengthen a patient's stride length,which can be achieved by slowing down the imagery's motion speedparameter. By modifying the visual scenery, the patient would be drivento model the visual example presented to them using a mirroringmechanism of action.

It is important to understand that the audio and visual outputs aremutually reinforcing stimuli: the visual scenery is layered together insynchrony to the rhythmic stimulus. Depending on the program selected(e.g., the CTA), the CTA module 3208 makes dynamic adjustments to thevisual scenery and rhythmic tempo in order to meet the therapy goal.

The foregoing example illustrates how the ANR system 3200 using the CTAmodule 3208 can control the synchronization of music tempo and ARscenery based on biomechanical sensor inputs in furtherance of gaittraining. It should be understood that the principles of this embodimentare applicable to many disease indications and rehabilitation scenarios.

In an exemplary configuration, the virtual treadmill can be generated bythe ANR system 3200 to modulate the patient's walking towards a targetparameter of oxygen consumption. In this example, the virtual treadmillis generated and controlled in order to modulate the walking speedtowards an oxygen consumption or efficiency target parameter usingentrainment. FIG. 33 is a graphical visualization of a real time sessionperformed using the ANR system 3200 with V02 max as target parameter,tempo changes used as interim goal, and entrainment used to drive thephysiological changes related to v02 max. FIG. 33 shows an example ofhow this process works in real time. More specifically, FIG. 33 is agraphical user interface illustrating various salient data-points andparameter values that are measured, calculated and/or adjusted by theANR system in real-time during a session. As shown, the top of theinterface shows a chart of entrainment potential values calculated foreach step in real-time throughout the session. In particular, the topbar shows individual EP calculated per step, which in this example isthe phase correlation between step time intervals and beat timeintervals. The central zone around EP=1 represents steps that aresufficiently entrained to the beat, or in other words, steps having anEP value within a prescribed range). The next window down provides astatus bar showing whether parameters are within a safe range. The nextwindow down shows a real time response driven by the CTA based on, interalia, the measured parameters, entrainment and other aforementionedinputs and feedback to the CTA. In particular, the circle iconsrepresent algorithm responses, which include both the tempo changes andrhythmic stimulus level (e.g. volume) changes. The bar below that showsjust the tempo and tempo changes by themselves. The next window downshows the real-time tempo of rhythmic stimulus provided to the patientover time in accordance with the CTA response. The bottom window showsmeasured oxygen consumption over time and target parameter. Although notshown in FIG. 33, it should be understood that the patient can bepresented with an augmented reality scene (e.g., the virtual treadmill)with visual elements animated in synchrony to the rhythmic stimulus anddynamically adjusted in synchrony the with the adjustments to the realtime tempo of the rhythmic stimulus. An example of how the AA/VR module3210 can be configured to synchronize the visual animation speed andaudio can include defining the relationship between displayed repetitivemotion rates and the tempo of the audio cues. For example, based on thebeat tempo, the rate of the treadmill and spacing of the steps arecalculated to define the relationships between audio and visualelements. Furthermore, a reference position of the treadmill, timing ofthe footsteps and any beat-timed animations are synchronized to theoutput time of the beats comprising the beat tempo. Using time scalingand video frame interpolation techniques known the animation industry, awide range of synchronized virtual scenes can be programmaticallygenerated by the AA/VR module 3210 on demand according to the definedrelationships between audio and visual elements.

FIG. 34 is a graph depicting Metabolic change during a first trainingsession for 7 patients (denoted by respective sets of two dots connectedby a dashed line). FIG. 34 shows data that supports purposely entrainingcan improve the oxygen consumption of an individual. In this instance,the graph shows a person's oxygen consumption (ml of oxygen/kg/meter)pre-training to rhythm and post-training to rhythm with the ANR system.This figure shows an average reduction of 10%. The results indicate thatthe entrainment process can improve endurance and reduce energyexpenditure while walking.

This foregoing process can be similarly implemented for each of thevarious possible target parameters mentioned above (e.g., oxygenconsumption could be exchanged for an alternative goal such as heartrate) and can be performed for walking or other interventions discussedin connection with FIGS. 18 through 31.

In accordance with one or more embodiments, the ANR system 3200 can beconfigured to compare real-time measured information concerningmovements of a person to AR images and/or components of music content(e.g. instantaneous tempo, rhythm, harmony, melody, etc) being output toa patient during the therapy session. This can be used by the system tocalculate an entrainment parameter, determine phase entrainment, orestablish baseline and carryover characteristics. For example, the ARimages could be moving at the same speed or cadence as the targetparameter. Alternatively, the AR relevant movements of the images couldbe entrained to the rhythmic stimulus in synchrony with how the personshould be moving.

An example of an AR 3-D dynamic model output can include projecting atherapist (virtual actor) or person (virtual actor) walking in thepatient's field of view which is initiated by the person performing thetherapy (real therapist). FIG. 35, for instance illustrates the view ofa therapist or coach projected in front of patient or trainer via AR,using for example AR glasses known in the art. This AR 3-D dynamic modelis controlled with one or a variety of CTA's. In a combination of CTA's,the virtual therapist could start with the approach shown and describedin connection with FIG. 22, and then have them proceed with a gaittraining regimen, like shown and described in connection with FIG. 18.Alternatively, these could be done simultaneously with dual tasking.During these instances, the virtual actor can be controllably displayedby the system as walking or moving backwards or forward with a smoothmovement similar to the non-affected side of the patient. This couldactivate mirror neurons, which is where affected neurons are“encouraged” to mirror in performance non affected neurons. This processcan also include providing an audio stimulus to sync the virtual and/orphysical person to the stimulus.

In another example, the AR 3-D dynamic model can be configured tosimulate a scenario in which the patient is walking in or around a crowdof people and/or people with objects in front and/or the side of thepatient. FIG. 36A, for example, illustrates the view of a crowd ofpeople projected in front of the patient via AR. The system can beconfigured to project the crowd or person traveling faster or slowerthan the baseline of the person to encourage them to move at a similarspeed or stopping/starting in a real-world environment. The crowd orperson could be entrained to the beats of the rhythmic auditory stimulusor another desired goal. Varying levels of difficulty in navigation canbe initiated by the AR 3-D dynamic model. As should be understood, theAR view of a therapist, crowd, person, obstacles and the like can bedynamically adjusted using the AR 3-D dynamic model according to theoutput of the CTA's.

In another example, the AR 3-D dynamic model can be configured tosimulate a scenario in which the patient is walking in or around anarrangement of cones which implements a virtual obstacle course for thepatient to navigate. Cones are a normal obstacle in a therapyenvironment, however other embodiments of this could be configured tosimulate normal activities of daily living (e.g., grocery shopping).These cones, along with virtual obstacles can encourage directionchanges by virtue of walking with side steps to each side and walkingbackwards, rather than just forward walking directional changes. Heretoo, wireless beacon triggers can be used to cause the ANR system topresent cones that appear and/or disappear. The beacons would betriggered based on detecting the location of the person related to thecones. In addition, varying levels of difficulty in navigation time andlength can be initiated. The target parameter for this example can be ameasure of walking speed or walking quality. Successful navigation wouldbe to navigate around the cones without virtually hitting them. Thesystem can be configured to present levels that get more difficult (e.g.more obstacles and faster speeds) as long as the person is successfullyavoiding the obstacles and the quality of walking does not degrade (asmeasured by increase in variability or worsening asymmetry).

In another example, the AR 3-D dynamic model can be configured tosimulate a scenario in which the patient is walking in or around cavesand/or cliffs which can include obstacles for a reality effect. Therealism would heighten details required for navigation over the priorpresented use cases. In another example with a person with anasymmetrical gait pattern, a winding path can be presented where itrequires a person to take a longer step on their affected side. Thiswinding path could also be separate cliffs that they have to step over avalley to not fall off. Wireless beacon triggers can be used to causethe ANR system to make cave and/or cliff obstacles appear and/ordisappear, thus varying levels of difficulty in navigation times andpath lengths. Sensor data can be used by the system to sync movements tothe winding path. The navigation requirements by the patient could bebiomechanical responses for navigating changes in a baseline prescribedcourse. The system is configured such that wireless spatial and temporalbeacon triggers affect the changes in the AR 3-D dynamic model. Thetemporal aspect of these wireless triggers is the ability to turn themon and off. This would allow for maximum flexibility in scriptingnavigation paths for the courses that patients should take as part ofthe therapy sessions. The target parameter for this instant is a measureof walking speed or walking quality. Successful navigation would be tonavigate the paths without stepping off the path or falling off thecliff. The system can be configured to present levels that would getmore difficult (e.g. more obstacles and faster speeds) as long as theperson is successfully staying on the path and the quality of walkingdoes not degrade (as measured by increase in variability or worseningasymmetry).

In another example, the AR 3-D dynamic model can be configured tosimulate a scenario in which the patient is standing or seatedstationary and asked to march as a virtual object is presented andapproaches each foot. FIGS. 36B, for example, illustrates the view offoot prints projected in front of patient via AR. The ANR system cangenerate a virtual scene in which the object may approach to the left orright of the patient to encourage side stepping. The object will bepresented as approaching the patient at a pre-defined tempo or beatwhich will follow a decision tree as described in FIG. 22. A visual ofthe correct movement by therapist or patient from past therapy may alsobe projected.

In another exemplary AR 3-D dynamic model implementation, the ANR systemcan be configured to incorporate haptic feedback into the therapy.Haptic feedback, for example, can be employed by the system as a signalif the user gets too close to objects or people in the projected ARsurrounding. Rhythmic haptic feedback may also be synced with theauditory cue to amplify sensory input. AR may also be adaptively andindividually enabled to cue initiation of movement, for example, duringa freezing of gait episode in someone with Parkinson's Disease.

In another exemplary AR 3-D dynamic model implementation, the ANR systemcan be further configured to incorporate optical and head tracking. Thistracking may be incorporated as feedback to the ANR system configured totrigger auditory input in response to where their eyes or head isfacing. For example, someone with left neglect who is interacting withonly the right side of their environment, the eye and head tracking canprovide input into how much of their left hemisphere environment isbeing engaged and trigger the system to generate an auditory cue todrive more attention to the left hemisphere. This data can also be usedto track progress over time, as clinical improvement can be measured bydegrees of awareness in each hemisphere. Another example of this is withpeople who have ocular motor disorders, where visual scanning from leftto right may be improved by doing it to an external auditory rhythm.

In another exemplary AR 3-D dynamic model implementation, the ANR systemcan be configured to provide a digital presence of past sessions todisplay a user's improvement. These models could be replayed after asession to compare from session to session or the lifetime of thetreatment. The digital presence of past sessions (or augmented session),when paired with the audio input of that session, could be used as amental imagery task for practice in between walking sessions and limitfatigue. The model would display differences in walking speed, cadence,stride length, and symmetry to help show the users changes over time andhow the treatment may be improving their gait. This presence could alsobe used by therapists before a session to help prepare training plans ortechniques for follow-on sessions. This modeled presence could also beused by researchers and clinicians to better visualize and reanimate in3-D imagery the evolution of a patient's progress.

In another exemplary AR 3-D dynamic model implementation, The AR/VRenvironments synced with the music content could create differentwalking or dance patterns to include ovals, spirals, serpentines,crossing paths with others, and dual task walking. Dance rhythms such asa Tango have been shown to have benefits stemming from Neurologic MusicTherapy (NMT) and RAS that can apply to the entire human body.

In accordance with one or more embodiments, the ANR system can beconfigured to utilize AA techniques to enhance the entrainment process,provide environmental context to a person, and aid in the AR experience.To enhance the recovery process, the system can be configured togenerate the exemplary AA experiences further described herein based oninputs taken from the environment, sensor data, AR environments,entrainment, and other methods.

An example of AA of a therapy/medical use case would be to addresssafety concerns and mitigate risk to patients who are performing therapyexercises. The ANR system can be configured to improve situationalawareness while listening to music with headphones by mixing externalsounds that exceeds a minimum audio loudness threshold instantaneouslyinto the therapy's rhythmic and audio cueing content. An example of anexternal sound would be the honking of a car or the sirens of anemergency vehicle, which would in synchrony automatically interrupt thenormal auditory stimulus to provide awareness to the person as to thepotential for danger. To perform this function and other functions, thelistening apparatus could have additional microphones and digital signalprocessing dedicated to performing this task.

In a further embodiment, the ANR system implementing AA can beconfigured to combine aspects of AA and the manipulation of spatialperception by aligning the rhythmic auditory cueing with a patient's“affected side” while they are performing a walking therapy session. If,for example, the right side of the patient requires a greater degree oftherapy, the audio cueing content can be spatially aligned with theright side for emphasis. Exemplary systems and methods for neurologicrehabilitation using techniques for side-specific rhythmic auditorystimulus are disclosed in co-pending and commonly assigned U.S. PatentApplication Number 62/934,457, titled SYSTEMS AND METHODS FOR NEUROLOGICREHABILITATION, filed on Nov. 12, 2019, which is hereby incorporated byreference as if set forth in its entirety herein.

In a further embodiment, the ANR system implementing AA can beconfigured to provide unique auditory cueing to increase spatialawareness of head position while gait training, encouraging the user tokeep head up, at midline and eyes forward, improving balance and spatialawareness while going through an entrainment process or other CTAexperience.

In a further embodiment, the ANR system implementing AA can beconfigured to provide binaural beat sound and tie it into humanphysiology (e.g. breathing rate, electrical brain activity (EEG) andheart rate) to improve cognition and enhance memory. The ANR system canbe configured to provide the binaural beat audio signal input incomplement to RAS signal input. The real-time entrainment and quality ofgait measurements being made by the system would likewise becomplemented by physiological measurements. For instance, the system asconfigured for binaural beat audio uses differential frequency signalsoutput in the left and right ears, whose difference is 40 Hz—the “Gamma”frequency of neural oscillation. These frequencies can reduce amyloidbuildup in Alzheimer's patients and can help with cognitive flexibility.By delivering such audio signal via the AA device to the user while theyperform RAS gait training, a second type of neural entrainment can beachieved simultaneously with the biomechanical-RAS entrainment. Thenetwork hypothesis of brain activation implies that both walking andcognition would be impacted. Such auditory sensory stimulation wouldtherefore entrain neural oscillations in the brain while rhythmicauditory stimulation entrains the motor system.

In a further embodiment, the ANR system implementing AA can beconfigured to provide a phase coherent soundstage (e.g. the correctaudio spatial perspective) when a patient rotates their head or changesits attitude. A sound stage is the imaginary 3-D image created by stereospeakers or headphones. It allows the listener to accurately hear thelocation of sound sources. An example of manipulating the soundstage ina therapeutic session would be keeping the voice sound of a virtualcoach “in-front” of the patient, even while their head may be turned tothe side. This feature could help avoid disorientation, thus creating amore stable, predictable and safe audio experience while performing thetherapy. This feature could be combined with the AR virtualcoach/therapist in front of a person in FIG. 34. It could also becombined with knowledge of the course or the direction the person needsto take in the real world.

In a further embodiment, the ANR system can be configured to combine AAwith Augmented Reality (AR) in such a manner that as the patientsynchronizes with a virtual crowd, the virtual sound effects (e.g.encouragements and crowd footsteps) create a coherent soundstage withregard to the patient's visual gaze. The audio could also create aperception of distance from, or nearness to, an object. Changing spatiallocation or loudness in such manner could also be used as a goal targetin combination with AR and 3-D imagery.

In a further embodiment, the ANR system can be configured to combine AAwith Augmented Reality (AR) in such a way to create virtual instrumenttherapy. Instruments such as bells, drums, piano, and guitar can becommon training tools for patients. By creating digital models of theseinstruments and providing an AA feedback upon interaction, the patientcan be given an immersive experience and the perception that they arephysically playing an instrument. This could be modified for difficultyto help the progression of a patient over time and show improvements.Examples of modifications could include adding more keys on a piano ormore strings on a guitar. In addition to virtual instruments, virtualsheet music or musical notation could be displayed in real time as thepatient is playing the instruments, either virtual instruments or realinstruments. Other examples could be in combination with the conceptsdiscussed in connection with FIG. 19, wherein the connected hardwarecould be replaced by AR. Similar logic could be used to other of thedocumented interventions.

In a further embodiment, the ANR system can be configured to implementAA in combination with a telepresence to provide a spatially accurateaudio experience for the therapist. The audio could also be generated tocreate a perception of distance from, or nearness to, an object. Bychanging spatial location or loudness of the AA and leveraging the ARmodel, the system can be used to determine more effectively if thepatient is meeting goals associated with playing the virtual devices andprovide them with a more accurate special experience.

In accordance with one or more embodiments, the ANR system can implementa type of AA, namely, a Rhythmic Stimulus Engine (RSE). The rhythmicstimulus engine is a bespoke rhythmic auditory stimulus, which embodiesthe principles of entrainment to drive a therapeutic benefit whilegenerating original and custom auditory rhythmic content for thepatient. For some disease states such as Parkinson's, it could also bebeneficial to have a constant rhythm “soundtrack” in the patient'senvironment. An RSE could be configured to perform this continuousbackground rhythmic neurostimulation without the need to accesspre-recorded music. In one example, the ANR system can be configured toimplement AA in combination with the Rhythmic Stimulus Engine (RSE) andAR to create a completely synchronized feedback state between incomingbiometric data, external audio inputs from the therapy environment, tothe generated rhythmic content, AR and AA outputs. In another example,the system can be configured to modulate the tempo of the rhythmic audiocontent generated by an RSE by the walking cadence of the patient in aninteractive fashion. In another example, the tempo and time signature ofthe rhythmic audio content generated by the RSE could be modulated bythe entrainment precision and beat factor of the patient user, such asone using a cane or assistive device in an interactive fashion. Inanother example, an RSE could provide the neurostimulation that, incombination with assistive technologies such as exo-suits, exo-skeletonsand/or FES devices, increases the effectiveness of walking therapy. Inanother example, an RSE could generate from a stored library oftraditional dance rhythm templates, the rhythmic audio content thatcould extend therapy to the patient's upper body and limbs. This couldbe extended to combine with AR techniques mentioned above, such as adancing crowd or virtual dancefloor. In another example, machinelearning techniques such as self-learning AI and/or a rules-based systemcould generate rhythm in real-time moderated by inertial motion units(IMUs) inputs that report cadence and quality of gait parameters, suchas symmetry and gait cycle time variability. Using an unsupervised MLclustering or decision-tree model, various gait patterns could act asinputs to the generative music system.

In accordance with one or more embodiments, the ANR system can implementa type of AA, namely, Sonification, which means applying varying amountsof signal distortion to the music content depending on how close to orfar from a patient is to a target goal. The degree and type ofsonification helps nudge the patient to a correction state. The novelcombination of sonification and entrainment could provide a feed-forwardmechanism for auditory motor synchrony through entrainment, whilesimultaneously providing a feedback mechanism via distortion of themusic content of some other biomechanical or physiological parameterthat the individual can adjust. For example, adding signal distortion tothe music signal while increasing the volume of the rhythmic cueingcould in combination have greater effectiveness than either method byitself.

In accordance with one or more embodiments, the ANR system can implementCTA in combination with a neurotoxin injection is as follows. The CTAcould apply the entrainment principle to work towards improving a motorfunction, such as gait. The neurotoxin injection can also be used totarget gait improvements by targeting spasticity in the muscles. Theseinjections take 2-4 days to take effect and last up to 90 days of effect(e.g. effectiveness period). The dosing of the CTA for entrainmentprinciples (e.g., the setting of one or more parameters of the CTA)could be targeted towards the effectiveness curve of the neurotoxininjection, where training is done less intense in the period before theinjection takes effect and increases during the effectiveness period.

In accordance with one or more embodiments, the ANR system can beconfigured to calculate entrainment parameter using the syncing of theheartbeat or the respiration rate to the music content, instead of thebiomechanical movement parameter. An example of a use case is for peoplewith agitation from various forms of dementia, Alzheimer's, bi-polardisorder, schizophrenia, etc. In this use case, the baseline parametercould be determined by the heart rate or respiration rate. Theentrainment or phased entrainment could be determined by the comparisonof the music content to the heart rate or respiration. Additionally,goals could be set to lower the amount of agitation to enhance thequality of life of these people.

As can be appreciated from the foregoing discussion, systems foraugmented neurologic rehabilitation of a patient in accordance with oneor more embodiments of the disclosure can comprise one or more of thefollowing points:

A computing system for having one or more physical processors configuredby software modules comprising machine-readable instructions. Thesoftware modules can include a 3D AR modelling module that, whenexecuted by the processor, configures the processor to generate andpresent augmented-reality visual and audio content to a patient during atherapy session. The content includes visual elements moving in aprescribed spatial and temporal sequence and rhythmic audio elementsoutput at a beat tempo.

The computing system also includes an input interface in communicationwith the processor for receiving inputs including time-stampedbiomechanical data of the patient relating to the movements performed bythe patient in relation to the AR visual and audio content andphysiological parameters measured using one or more sensors associatedwith the patient.

The software modules also include a critical thinking algorithm thatconfigures the processor to analyze the time-stamped biomechanical datato determine a spatial and temporal relationship of the patient'smovements relative to the visual and audio elements and determine alevel of entrainment of the patient relative to a target physiologicalparameter. Additionally, the 3D AR modelling module further configuresthe processor to dynamically adjust the augmented-reality visual andaudio content output to the patient based on the determined level ofentrainment relative to the target parameter.

The above systems, devices, methods, processes, and the like may berealized in hardware, software, or any combination of these suitable foran application. The hardware may include a general-purpose computerand/or dedicated computing device. This includes realization in one ormore microprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable devices orprocessing circuitry, along with internal and/or external memory. Thismay also, or instead, include one or more application specificintegrated circuits, programmable gate arrays, programmable array logiccomponents, or any other device or devices that may be configured toprocess electronic signals. It will further be appreciated that arealization of the processes or devices described above may includecomputer-executable code created using a structured programming languagesuch as C, an object oriented programming language such as C++, or anyother high-level or low-level programming language (including assemblylanguages, hardware description languages, and database programminglanguages and technologies) that may be stored, compiled or interpretedto run on one of the above devices, as well as heterogeneouscombinations of processors, processor architectures, or combinations ofdifferent hardware and software. In another aspect, the methods may beembodied in systems that perform the steps thereof, and may bedistributed across devices in several ways. At the same time, processingmay be distributed across devices such as the various systems describedabove, or all the functionality may be integrated into a dedicated,standalone device or other hardware. In another aspect, means forperforming the steps associated with the processes described above mayinclude any of the hardware and/or software described above. All suchpermutations and combinations are intended to fall within the scope ofthe present disclosure.

Embodiments disclosed herein may include computer program productscomprising computer-executable code or computer-usable code that, whenexecuting on one or more computing devices, performs any and/or all thesteps thereof. The code may be stored in a non-transitory fashion in acomputer memory, which may be a memory from which the program executes(such as random access memory associated with a processor), or a storagedevice such as a disk drive, flash memory or any other optical,electromagnetic, magnetic, infrared or other device or combination ofdevices. In another aspect, any of the systems and methods describedabove may be embodied in any suitable transmission or propagation mediumcarrying computer-executable code and/or any inputs or outputs fromsame.

It will be appreciated that the devices, systems, and methods describedabove are set forth by way of example and not of limitation. Absent anexplicit indication to the contrary, the disclosed steps may bemodified, supplemented, omitted, and/or re-ordered without departingfrom the scope of this disclosure. Numerous variations, additions,omissions, and other modifications will be apparent to one of ordinaryskill in the art. In addition, the order or presentation of method stepsin the description and drawings above is not intended to require thisorder of performing the recited steps unless an order is expresslyrequired or otherwise clear from the context.

The method steps of the implementations described herein are intended toinclude any suitable method of causing such method steps to beperformed, consistent with the patentability of the following claims,unless a different meaning is expressly provided or otherwise clear fromthe context. So, for example performing the step of X includes anysuitable method for causing another party such as a remote user, aremote processing resource (e.g., a server or cloud computer) or amachine to perform the step of X. Similarly, performing steps X, Y and Zmay include any method of directing or controlling any combination ofsuch other individuals or resources to perform steps X, Y and Z toobtain the benefit of such steps. Thus, method steps of theimplementations described herein are intended to include any suitablemethod of causing one or more other parties or entities to perform thesteps, consistent with the patentability of the following claims, unlessa different meaning is expressly provided or otherwise clear from thecontext. Such parties or entities need not be under the direction orcontrol of any other party or entity, and need not be located within aparticular jurisdiction.

It should further be appreciated that the methods above are provided byway of example. Absent an explicit indication to the contrary, thedisclosed steps may be modified, supplemented, omitted, and/orre-ordered without departing from the scope of this disclosure.

It will be appreciated that the methods and systems described above areset forth by way of example and not of limitation. Numerous variations,additions, omissions, and other modifications will be apparent to one ofordinary skill in the art. In addition, the order or presentation ofmethod steps in the description and drawings above is not intended torequire this order of performing the recited steps unless a particularorder is expressly required or otherwise clear from the context. Thus,while particular embodiments have been shown and described, it will beapparent to those skilled in the art that various changes andmodifications in form and details may be made therein without departingfrom the spirit and scope of this disclosure.

What is claimed is:
 1. A system for augmented neurologic rehabilitationof a patient: a computing system having a processor configured bysoftware modules comprising machine-readable instructions stored in anon-transitory storage medium, the software modules including: an AA/ARmodelling module that, when executed by the processor, configures theprocessor to generate an augmented-reality (AR) visual content andrhythmic auditory stimulus (RAS) for output to a patient during atherapy session, wherein the RAS comprises beat signals output at a beattempo and wherein the AR visual content includes visual elements movingin a prescribed spatial and temporal sequence based on the beat tempo;an input interface in communication with the processor for receivingreal-time patient data including time-stamped biomechanical data of thepatient relating to repetitive movements performed by the patient intime with the AR visual content and RAS, and wherein the biomechanicaldata is measured using a sensor associated with the patient; and thesoftware modules further including: a critical thinking algorithm (CTA)module that configures the processor to analyze the time-stampedbiomechanical data to determine a temporal relationship of the patient'srepetitive movements relative to the visual elements and beat signalsoutput at the beat tempo to determine a level of entrainment relative toa target parameter; wherein the AA/AR modelling module furtherconfigures the processor to dynamically adjust the AR visual and RASoutput to the patient in synchrony and based on the determined level ofentrainment.
 2. The system of claim 1, wherein the processor dynamicallyadjusts the AR visual content and RAS based on the determined level ofentrainment by, adjusting the beat tempo of the RAS in view of the levelof entrainment, and adjusting the prescribed spatial and temporalsequence of the visual elements in synchrony with the adjusted beattempo.
 3. The system of claim 2, wherein the beat signals are eachoutput at a respective output time; wherein the level of entrainment isdetermined based on a timing of the repetitive movements relative to therespective output times of the beat signals.
 4. The system of claim 3,wherein the CTA module configures the processor to determine the levelof entrainment by: analyzing the time-stamped biomechanical data toidentify a respective time of respective repetitive movements, measuringa temporal relationship between the respective time of one or morerepetitive movements and a respective output time of one or moreassociated beat signals, calculating an entrainment potential based onthe measured temporal relationship for one or more of the respectiverepetitive movements; and wherein the processor is configured todynamically adjust one or more of the AR visual and RAS output based onthe entrainment potential.
 5. The system of claim 2, wherein the CTAmodule further configures the processor to determine whether thebio-mechanical data or physiological data measured for the patient meetsa training goal parameter; wherein the AA/AR modelling module furtherconfigures the processor to dynamically adjust the AR visual content andRAS output to the patient in response to the training goal parameterbeing unmet.
 6. The system of claim 5, wherein the target parameter isthe beat tempo and wherein the training goal parameter is arehabilitation outcome.
 7. The system of claim 1, further comprising: anAR video output device configured to present the AR visual content tothe patient; an audio output device configured to output the RAS to thepatient; and the sensor associated with the patient and configured tomeasure the time-stamped biomechanical data of the patient, wherein thesensor comprises an inertial measurement unit (IMU) device.
 8. Thesystem of claim 5, further comprising: a sensor associated with thepatient and configured to measure physiological data of the patient, andwherein the training goal parameter is a physiological parameter, andwherein the physiological parameter is one or more of: heartrate, bloodoxygenation, respiratory rate, VO2, electrical brain activity (EEG). 9.The system of claim 1, wherein the AR visual content comprises a visualscene including one or more of: a virtual treadmill animated to appearas if a top surface of the treadmill is approaching the patient at arate that corresponds to the beat tempo, and a plurality of footprintssuperimposed on a top surface of the virtual treadmill, wherein thefootprints are arranged spatially and appear to approach the patient ata rate that corresponds to the beat tempo, and an animated personperforming the repetitive motion at a rate that corresponds to the beattempo.
 10. The system of claim 1, wherein modifying the AR visualcontent comprises one or more of: changing a spacing of the footprintsin view of a change in the beat tempo, changing the rate at which theanimated person performs the repetitive motion, changing the rate atwhich the virtual top surface of the treadmill appears to be approachingthe patient, changing the rate at which virtual obstacles or sceneperturbations appear to the patient.
 11. A method for augmentedneurologic rehabilitation of a patient having a physical impairment, themethod being implemented on a computer system having a physicalprocessor configured by machine-readable instructions which, whenexecuted, perform the method, comprising: providing rhythmic auditorystimulus (RAS) for output to a patient via an audio output device duringa therapy session, wherein the RAS comprises beat signals output at abeat tempo; generating augmented-reality (AR) visual content for outputto a patient via an AR display device, wherein the AR visual contentincludes visual elements moving in a prescribed spatial and temporalsequence based on the beat tempo and output in synchrony with the RAS;instructing, the patient to perform repetitive movements in time withthe beat signals of the RAS and corresponding movement of the visualelements of the AR visual content; receiving real-time patient dataincluding time-stamped biomechanical data of the patient relating torepetitive movements performed by the patient in time with the AR visualcontent and RAS, and wherein the biomechanical data is measured using asensor associated with the patient; analyze the time-stampedbiomechanical data to determine a temporal relationship of the patient'srepetitive movements relative to the visual elements and beat signalsoutput according to the beat signal to determine an entrainmentpotential; dynamically adjusting the AR visual content and RAS foroutput to the patient in synchrony and based on the determinedentrainment potential not meeting a prescribed entrainment potential;continuing the therapy session using the adjusted AR visual content andRAS.
 12. The method of claim 11, further comprising: measuring thebiomechanical data from the patient, wherein measuring the biomechanicaldata from the patient comprises providing a sensor associated with thepatient and measuring one or more of motion, acceleration and pressureassociated with movement of the patient.
 13. The method of claim 11,wherein dynamically adjusting the AR visual content and RAS based on thedetermined entrainment potential includes, adjusting the beat tempo ofthe RAS based on the entrainment potential, and adjusting the prescribedspatial and temporal sequence of the visual elements in synchrony withthe adjusted beat tempo.
 14. The method of claim 13, further comprising:comparing the beat tempo to a training goal parameter comprising a goalbeat tempo; and dynamically adjusting the RAS and the AR visual contentas a function of the comparison and the entrainment potential determinedin regard to the beat tempo.
 15. The method of claim 14, furthercomprising, if the entrainment potential meets a prescribed level andthe beat tempo is below the goal beat tempo, increasing the beat tempoof the RAS towards the goal beat tempo and adjusting the prescribedspatial and temporal sequence of the visual elements in synchrony withthe adjusted beat tempo.
 16. The method of claim 11, wherein the beatsignals are each output at a respective output time according to thebeat tempo, wherein measuring the entrainment potential comprisescomparing a respective onset time of each of a plurality of repetitivemovements with the respective output-time of the associated beat signalsin real-time, and wherein the onset time for a given repetitive movementis a time at which a prescribed identifiable event occurs during thegiven repetitive movement.
 17. The method of claim 11, furthercomprising: determining whether patient data comprising thebio-mechanical data or physiological data measured for the patient meetsa training goal parameter; wherein the AA/AR modelling module furtherconfigures the processor to dynamically adjust the AR visual content andRAS output to the patient in response to the training goal parameterbeing unmet.
 18. The method of claim 17, further comprising: measuringthe physiological data from the patient, wherein measuring thephysiological data from the patient comprises providing a sensorassociated with the patient configured to measure a physiologicalparameter, wherein the physiological parameter is selected from thegroup consisting of heart rate, respiratory rate and VO2 max.
 19. Themethod of claim 11, wherein the AR visual content comprises a visualscene including one or more of: a virtual treadmill animated to appearas if a top surface of the treadmill is approaching the patient at arate that corresponds to the beat tempo, and a plurality of footprintssuperimposed on a top surface of the virtual treadmill, wherein thefootprints are arranged spatially and appear to approach the patient ata rate that corresponds to the beat tempo, and an animated personperforming the repetitive motion at a rate that corresponds to the beattempo.